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Abstract 


This book offers a unique analysis of the use of the automated fa- 
cial recognition algorithms that are increasingly intervening in our 
society, from a critical visual culture studies perspective. The first 
part of this study traces the history of the merging of statistics and 
vision, examining the example of an early facial recognition algo- 
rithm called “eigenface,” while the second part addresses contem- 
porary artistic engagements with facial recognition technology in 
the work of Thomas Ruff, Zach Blas and Trevor Paglen. This book 
argues that we must take a closer look at automated facial recog- 
nition and claims that this technology is embedded in historical 
practices of visuality. Even more significantly, this technology, the 
book argues, is redefining what it means to see and be seen in the 
contemporary world. 
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Chapter 1: Introduction 


On Machinic Ways of Seeing the Face: Aim and Background 


While passing through border control at Helsinki airport on a flight 

to Seoul, I walked through a biometric turnstile. On a small screen 

by the entrance to the turnstile were instructions to scan my pass- 
port. Once my passport had been scanned, I was then allowed to 

enter a small cubicle space between two panes of glass. I stood fac- 
ing an empty screen that moved vertically, adjusting automatically 

to my height. A sensor took my picture, which then appeared be- 
fore me on the screen. I waited a moment while the machine ver- 
ified my identity by checking the facial image it had just captured 

against the image scanned from my passport. Then one of the glass 

panes opened, and I was released from the turnstile. I was then 

confronted with a border control guard - a man who sat behind a 

desk and whose face was shielded by a dark pane of glass. I only saw 

his hands, which grasped my documents. He did not seem to look 
at my face, but rather only examined my plane ticket and passport, 
which he eventually stamped with approval. My identity had been 

successfully recognized, and I was allowed to move on. 

Automated facial recognition (AFR) has increasingly become 

a filter through which access to the world is granted. The ritual to 

which I was subjected at Helsinki airport, a ritual I also experienced 
in a slightly altered form at the Swedish immigration office, as part 
of the process of registering for permanent residency, is a familiar 
one. Beyond its most conspicuous uses in border control and im- 
migration services, AFR is now increasingly being implemented in 
more mundane and everyday scenarios. For example, AFR systems 
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are used in our phones, at ATM machines, in office security systems, 
for the manning of cash registers in convenience stores and inside 
toilet-paper dispensers in public bathrooms. They are also used 
covertly, in CCTV and police cameras. In these new contexts, suc- 
cessful recognition by AFR is increasingly intervening in a complex 
negotiation between recognition, identity and access. Alongside 
this expansion of AFR into everyday contexts, there is a growing 
realization that we are becoming reliant on machines looking at us 
- and, most importantly, perceiving and interpreting us - and mak- 
ing decisions that, ultimately, govern our existence. 

Although AFR systems rely on a form of visual recognition, 
something about their processes is also paradoxically opposed to 
forms of looking, as is apparent in the example of the biometric 
turnstile at Helsinki airport given above. Often the operations of 
biometric scanning and identity verification occur within an invisi- 
ble field and through processes that the human subject does not see. 
As a result, algorithmic processes of biometric recognition remain 
difficult to define, analyze and critique. A close analysis of the tech- 
nology itself is necessary in order to understand how and by what 
means biometric identification occurs. Kelly Gates explains that the 
facial recognition technologies used in biometric identification “are 
being developed to address a fundamental concern of modern soci- 
eties: the problem of ‘disembodied identities’ [...] the existence of vi- 
sual and textual representations of individuals that circulate inde- 
pendent of the physical bodies.” Yet this re-embodying of identities 
that occurs through a facial recognition operation occurs through a 
disembodied form of visual perception. 

This book investigates AFR technology through an inquiry into 
its visuality, that is, into how an AFR process encultures a way of 
seeing and, as such, can be understood as a contemporary mode 
of perception by machine. It focuses on the ‘recognition’ part of 
automated facial recognition. Facial recognition through an AFR 
system contains three inherent tensions, which structure the fol- 
lowing discussion. The first tension is that AFR technology is a form 


1 Kelly A. Gates, Our Biometric Future: Facial Recognition Technology and the 
Culture of Surveillance (New York: New York University Press, 2011), 12. 
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of machinic vision that is used to recognize us and yet, as I have 
mentioned, operates unseen and is, indeed, unrecognizable to those 
within its scope. In this way, AFR technology involves an invisible 
operation; its processes of recognition are inaccessible to us. Yet 
this operation increasingly governs people’s lives within various 
institutional contexts. The second tension has to do with the ques- 
tion of the continuities and discontinuities between machinic visu- 
al perception and human visual perception. Many methods of AFR 
are designed and inspired by human processes of cognition and 
recognition, but to what extent do automated recognition processes 
replicate our own perceptual processes? To the extent that they do 
replicate these processes, in what ways does this replication relate, 
in turn, to a discourse of visuality? A third tension involves the re- 
lationship between the processes of recognition in an AFR system 
and the kinds of knowledge it may produce. A successful operation 
of automated recognition results in the production of (often action- 
able) information about the identity of a subject. An understanding 
of these processes of recognition may afford us a broader under- 
standing of contemporary forms of identity production, and under- 
standing these forms of identity production may allow us, in turn, 
to construct alternatives to the AFR process. 

These tensions also suggest a critique of the notion of recogni- 
tion inherent in AFR systems. In critically examining the notion 
of recognition, I intend to operationalize the term and use it as a 
tool to analyze AFR systems. These three tensions prompt a dis- 
cussion of what recognition does and can possibly mean, and they 
problematize this notion. They lay out a general problematic of AFR 
technology as it relates to the context of its implementation and 
explored through the specificities of its technical processes. These 
tensions relate to broader issues of how machinic processes de- 
fine visual perception as recognition. These tensions also provide 
an outline a line of inquiry into the relationship, in an AFR process, 
between seeing and knowing, that is, between how this process 
comes to produce not only the filtered data of information but also 
the knowledge that is accrued through the process of algorithmic 
learning in relation to the recognized subject. 
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AFR technology is an example of the automatization of the la- 
bor of looking, which is taking place in a variety of contexts in the 
Information Age. The production of massive amounts of data from 
digital surveillance networks has made it that the ability to per- 
form this labor of looking sometimes outstrips human capacities. 
AFR technology is utilized as a way of interpreting data and deriv- 
ing meaning from it in order to produce “information,” the primary 
product of this technology. Machine vision, or the automation of vi- 
sual sense perception, has its origins in the controlled environment 
of the industrial factory, where it is used as a means of sorting and 
inspecting industrial parts and manufacturing flows. In the Infor- 
mation Age, machine vision technology has developed to be able to 
read measurable and quantifiable objects outside of the factory set- 
ting. In conjunction with the increasing use of surveillance systems, 
machine vision comes to focus on the processes and transactions 
relating to the flows of people in society. Similar to its use within 
the factory, biometrics, as one form of machine vision, functions in 
operations of inspection, that is, the sorting, selecting and surveil- 
ling of individuals in society. The biometric gaze turns toward the 
shifting, unstable and unruly forms of the body and reads these as 
if they were quantifiable industrial objects. 

Facial recognition technology calls our attention to the site of the 
face not only as a means of identification but as a marker of identity. 
Inherent in the distinction between identity and identification is a 
politics of the face. In describing the site of the face as a particular 
mechanism of the political imaginary, Jenny Edkins writes, “the 
face in itself is a politics [...] that reflects and inscribes a particular 
intersection of two regimes of signs: the signifying and subjecting 
regimes.” While the face can be measured, scanned and read like a 
sign, it is also expressive of the malleability and shape-shifting na- 
ture of the subjective experience of self. Faces are everywhere these 
days. In the so-called Age of the Selfie, one’s self-portrait temporally 
and spatially situates oneself within the channels of various social 
networks. The selfie is an expression of the face as an embodiment 
of the self, depicting one’s experience at a particular place and time. 


2 Jenny Edkins, Face Politics (London: Routledge, 2015), 4. 
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This may be why David Lyon calls on the face as an ethical starting 
point for a critical analysis of biometrics.’ He describes the site of 
the face as “resisting mere categorization” and claims its treatment 
is central to understanding how “one perceives the issues surround- 
ing the appropriate conditions of self-disclosure.* While the face 
is, as Lyon argues, expressive of an embodied social person and as 
such resists simple categorization, it is also a site of the body that 
is easily accessed, without consent, by biometric technologies. The 
prevalence of AFR technology is in part due to the fact that the face 
is a part of the body that tends to be visible in daily life and so is eas- 
ily captured by surveillance mechanisms and CCTV. For example, 
in the arguments made in favor of AFR use in the ongoing “War on 
Terror,” it has been stated that the only biometric data available on 
terrorists is their facial images.° 

It has been noted by many programmers and developers of 
AFR technology that it confronts its own special challenges when 
it comes to both the detection and the recognition of human faces. 
The face is quite unlike the sorts of things typically the subjects of 
machine vision — subjects that are usually finite, measurable and 
geometrically fixed. As Lyon suggests, the face resists mere cate- 
gorization not only through its expression of a subjective and em- 
bodied sense of identity but also physically, in its variability, mul- 
tilayered forms of expression and constantly changing form, which 
resists the reductive methods of recognition used in AFR systems. 
In short, the face is a part of the body that escapes singular recog- 
nition. As such, the face as an object (and subject) of AFR technol- 


3 David Lyon, “Surveillance as Social Sorting: Computer Codes and Mobile 
Bodies,” in Surveillance as Social Sorting: Privacy, Risk and Digital Discrimi- 
nation, ed. David Lyon (London: Routledge, 2003), 27-28. 

4 Lyon, “Surveillance as Social Sorting,” 27. 

5 AFRis also in high demand because the use of facial images for identifica- 
tion purposes has a long history, and so there is a pre-existing infrastruc- 
ture on which AFR can draw. Passport photos and criminal portraiture are 
examples of the use of the face and facial images as means of identification. 

6 John D. Woodward Jr., Biometrics: Facing Up To Terrorism (Santa Monica, 
CA: Rand Corporation, 2001), 8, https://www.rand.org/pubs/issue_papers/ 
1P218.html. 
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ogy provides a fascinating case study of machine vision, because it 
challenges the mechanisms of this technology both technically and 
theoretically. 

A growing number of artists are working with facial recognition 
technologies in order to explore, confront and articulate sociopolit- 
ical issues raised by the use of these technologies. Artists engaging 
with facial recognition technology are uniquely positioned to artic- 
ulate the visuality of this technology, and as such their works figure 
centrally in this study as objects of theoretical examination. Much 
of the development of AFR systems, and the surrounding discourse, 
has been guided by the aims of this technology in policing, military 
and market contexts. These artworks fill in a gap by providing a cul- 
tural translation of the technology, a translation that is often not 
discussed or explored by developers or programmers. Artistic en- 
gagements with facial recognition technology are able to articulate 
complex issues that AFR processes give rise to at the intersection 
of recognition, identity and representation. The strategies of artists 
engaging with facial recognition technology include appropriating 
the technology and decontextualizing its processes; in these ways, 
these artists allow for an engagement with this technology that not 
only problematizes its use but also imagines alternative outcomes 
of the technology and its processes. 

There are three primary research questions that guide this 
study: 


1) What is the process through which recognition is defined in the 
machinic form of vision used in an AFR method? 

2) What historical continuities (and discontinuities) can we identi- 
fy in these processes of recognition? 

3) How do artistic interventions with facial recognition technol- 
ogy respond to, articulate and confront the implications of the 
use of this technology? 


The general purpose of these questions is to achieve a broader un- 
derstanding of AFR technology within the sociopolitical and cultur- 
al contexts in which it is used. These questions focus on the general 
problematic and central principle of AFR technology by asking how 
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recognition can be defined through a technical process. Although 
these questions approach a general problematic of recognition in 
AFR technology, they also get at the specificities of how this tech- 
nology operates. These questions reflect on the technology itself, on 
how a technical process encultures a way of seeing through recog- 
nition. These questions also relate the ways in which the face is rep- 
resented through an AFR method to a discourse of visuality, name- 
ly, historical practices of facial representation and portraiture. This 
line of inquiry acknowledges that the forms of visuality involved in 
an automated recognition process are a central source of the knowl- 
edge this technology produces. The ways in which the face is repre- 
sented in this process of recognition are how an AFR method itself 
comes to know a face. In addition, through a process of successful 
recognition, an AFR method produces knowledge in the form of the 
identity and identification of the person being recognized. In refer- 
ring to examples of artistic interventions by contemporary artists, 
this analysis presents an inquiry into the visuality of automated fa- 
cial recognition and the ways in which meaning is both produced 
and made malleable by AFR technology. 


Cultural Analyses of Biometrics: Previous Scholarship 


Biometrics and, in particular, AFR technology have increasingly 
come to be applied within the contexts of risk mitigation and secu- 
rity practices for the identification and recognition of individuals. 
Biometrics utilizes advanced visual technologies, such as digital 
sensors, to scan, measure and capture parts of the body, their forms 
and surface patterns. The uniqueness of body parts, such as finger 
prints, an iris or a person’s face, allow these technologies to ascer- 
tain an individual's identity. The practice of using the body as a sign 
of identity has a long history that dates back to the mid-1800s, be- 
ginning with the practice of using a hand imprint to seal a contract, 
a practice that has a more recent analogue version in the collection 
of the fingerprints of grade-school students. In the 1980s, the term 
“biometrics” began to be used to describe the automated systems 
of human recognition then being developed; in the 1970s, the field 
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had been known as “automated personal identification.” What has 
changed with more recent developments in biometrics is that these 
practices are now being digitized, which includes the use of algo- 
rithms to read collected bodily data and the use of biometric tech- 
nologies in conjunction with growing and widespread networks of 
surveillance. Alongside the increasing use and continued develop- 
ment of contemporary biometrics, there has been a corresponding 
increase in critical academic scholarship in the humanities that has 
studied biometric and AFR technology through a cultural lens. In 
the last ten years, scholars from the fields of sociology, surveillance 
studies and media and communications have produced work that 
articulates the social, political and cultural implications of society’s 
increasing dependence on biometrics. The scholars included in this 
brief overview of previous literature for the most part critically 
approach the use of biometric and AFR technology through anal- 
yses of their technological development and the systemic contexts 
and implications of their implementation. They have analyzed the 
technology through theoretical frameworks that draw on a range 
of discourses, including sociology, surveillance studies, science and 
technology studies, post-colonial theory and gender and feminist 
theory. 

One of the first scholars to adopt a sociological approach to bio- 
metric technologies was David Lyon, who in 2001 described biomet- 
rics as one aspect of the growing ubiquity of widespread surveil- 
lance practices that function as a method of “social sorting,” that is, 
as a way of “categorizing populations and persons for risk assess- 
ment [...] [in] attempts to minimize risk, by discovering - preferably 
in advance — who is likely to break the law, buy the product, or seek 
the service.” Lyon’s use of the term “social sorting” highlights the 

“classifying drive” as a central function of surveillance and biomet- 


7 James L. Wayman, “The Scientific Development of Biometrics over the Last 
40 Years,” in The History of Information Security: A Comprehensive Handbook, 
ed. Karl de Leeuw and Jan Bergstra (Amsterdam: Elsevier, 2007), 263-74. 

8 David Lyon, “Facing the Future: Seeking Ethics for Everyday Surveil- 
lance,” Ethics and Information Technology 3, no. 3 (2001): 172, https://doi. 
org/10.1023/A:1012227629496. 
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ric technologies.’ Rather than raising issues of privacy concerning 
the individual, Lyon brings the discussion of biometric and surveil- 
lance practices into the social realm, examining their use as mech- 
anisms of neoliberal policy that enforce social division and catego- 
rization and imply unequal access and distribution. Lyon’s analyses 
of biometrics establish a framework through which to approach the 
social power of its information production. 

Lyon’s critique of biometrics as a surveillance practice high- 
lights its unequal implementation and effects on vulnerable and 
marginalized parts of the population. This topic is expanded on in 
the work of Simone Browne and Shoshana Magnet, both of whom 
analyze the dialectics of recognition in biometric technology and 
the ways it is directed by normative categories of identity. Both 
scholars the ways in which applications of biometric technology 
deny a subjectivity, and they associate the limitations of the tech- 
nology with a limit to notions of identity. Biometric technologies 
thus enact a negation of certain individuals based on their race, 
gender and/or economic status while verifying the identities of cer- 
tain others. Through the use of different theoretical frameworks 
and discourses, both Browne and Magnet recognize that there is a 
cultural logic embedded in biometric technology, and their works 
actively critique this logic. 

Inher book Dark Matters: On the Surveillance of Blackness, Browne 
draws on the discourse of post-colonial theory, referencing the work 
of Franz Fanon and his concept of “epidermalization” to approach 
the power dynamics of a biometric, disembodied gaze. She defines 
the concept of “digital epidermalization’” as “the exercise of power 
cast by the disembodied gaze of certain surveillance technologies 
[...] that can be employed to do the work of alienating the subject by 
producing a truth about the racial body and one’s identity (or iden- 
tities) despite the subject’s claims.” Browne’s analysis outlines his- 
torical continuities that can be found within biometric and surveil- 
lance practices by placing these in direct dialogue with an archive 


9 Lyon, “Surveillance as Social Sorting,” 13. 


10 Simone Browne, Dark Matters: On the Surveillance of Blackness (Durham, 
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of transatlantic slavery and the history of the control, regulation 
and surveillance of black bodies. Through her analysis, Browne de- 
scribes the inherent violence that occurs through the enactment of 
institutional recognition by the state. 

Magnet’s book When Biometrics Fail: Gender, Race and the Technol- 
ogy of Identity adopts a feminist theoretical framework." The point 
of departure for her study is a call for a broader and more precise 
vocabulary for defining a notion of failure in relation to biometric 
technology. In this way, Magnet aims to counter an acceptance of 
failure as technically productive: technical failures are usually ac- 
cepted as means to an end, leading to further developments and 
eventual success. Instead, Magnet defines a sociological framework 
of failure according to which, when biometrics technologies “over 
target” and “fail to identify,” these failures are defined as sociolog- 
ically counterproductive, as excluding certain communities and 
resulting in the inaccessibility of resources for certain segments 
of the population. Magnet argues that the failure of the science of 
biometrics lies in the fact that it is a technological implementation 
of gendered and racialized norms, essentially codifying “existing 
forms of discrimination”” and thereby failing to recognize the com- 
plexity of bodily identity. 

Magnet’s analysis may be seen alongside the work of other 
scholars who focus on the technical limitations of biometric tech- 
nologies. Kelly Gates’s book Our Biometric Future: Facial Recognition 
Technology and the Culture of Surveillance, in particular, challenges 
the legitimacy of AFR technologies by examining the gap between 
the claims made about their viability and their actual capabili- 
ties.” Drawing on communications theory, Gates gives an in-depth 
account of the political and economic constellation of influences 
governing the development and implementation of AFR technolo- 
gy. Gates argues that the claims of technical precision made on be- 
half of AFR technology by the biometrics industry and government 


11 Shoshana Magnet, When Biometrics Fail: Gender, Race and the Technology of 
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officials make its implementation appear inevitable. Furthermore, 
these claims mean that AFR technology is often given precedence 

over other forms of intelligence gathering. In examining the actual- 
ities and limitations of the technology, Gates’s study aims to disrupt 
this trajectory and to provide a more down-to-earth account of its 

advanced capabilities. 

Gates explores how AFR technology was framed as the solution 
to the particular challenges of the post-9/11 period. She describes the 
narrative advanced by the research agency of the US Department 
of Defense (DARPA): that the war on terror involved a new kind of 
enemy, an “unidentifiable” enemy who thus implied a new form of 
national vulnerability. Gates explains that, in virtue of AFR tech- 
nology’s ability to produce a kind of identifiability for these enemies, 
and thereby provide a sense of certainty in the post-9/11 geopolitical 
landscape, the need to deploy this “expensive, new, high-tech sur- 
veillance technology [...] seemed self-evident.” Gates’s study shows 
how the development of AFR is based on the fact that this technol- 
ogy has been framed as a solution to the political and military chal- 
lenges faced within the contemporary geopolitical landscape. 

Numerous scholars have approached an analysis of biomet- 
rics through the lens of geopolitics. One example is Btihaj Ajana, a 
scholar working in the area of digital cultures who approaches an 
study of biometrics through the theoretical discourse of biopolitics, 
drawing on the work of Michel Foucault, Giorgio Agamben and Ni- 
kolas Rose.” Ajana argues that biometric technologies enact a form 
of biopolitics and result in the construction of politicized identities 
along geopolitical lines. For example, Ajana examines the binary of 
the “asylum seeker” and the “neoliberal citizen,” which in turn com- 
prises a whole set of practices that govern the individual: hierar- 
chical power relations marginalize asylum seekers, trampling their 
basic rights, especially their rights to move freely, and also empow- 
er the neoliberal citizen, at the opposite end of the geopolitical spec- 


14 Ibid. 
15 Ibid. 
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25 


26 


Portraits of Automated Facial Recognition 


trum, with a “surplus of rights.” Louise Amoore, a scholar in the 
field of political geography, also approaches an analysis of biometric 
technology within the context of geopolitics. She does so by draw- 
ing on a conceptual framework from the field of geography. In her 
concept of the “biometric border,” she casts biometrics as a way of 
constructing a new kind of geopolitical border, which is made man- 
ifest through the mapping and measuring of the body." Biometrics 
does not analyze abstractions but maps the actual biophysical pat- 
terns of the body and, by doing so, inscribes the body with insti- 
tutional divisions and demarcations. As Amoore writes, “In effect, 
the biometric border is the portable border par excellence, carried 
by mobile bodies at the very same time as it is deployed to divide 
bodies at international boundaries, airports, railway stations, on 
subways or city streets, in the office or the neighbourhood.” 

The work of these scholars presents a range of diverse approach- 
es to the cultural, social and political implications of biometrics and 
AFR technology. These scholars articulate some of the most perti- 
nent and salient conclusions of the cultural and social critique of 
these technologies. They explore how the development and imple- 
mentation of these technologies depends on a certain way of fram- 
ing both the solution and the problem. Overall, these scholars’ works 
undermine the claims of neutrality and precision so often made on 
behalf of biometric and AFR technologies. These scholars recognize 
that biometrics and AFR technology directly participate in decision 
making, with wide-ranging social and political implications. They 
look into the social and economic interests behind the development 
of the technologies as well as the results in terms of contemporary 
constructions of institutional identities. These cultural analyses of 
biometric technologies recognize that the implementation of these 
technologies brings about not only a technological but a cultural 
shift. By bringing to bear a diverse range of theoretical discourses 
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in their discussions of biometric technology, these scholars articu- 
late some of the cultural implications of the implementation of this 

technology. A central critique of biometric technologies advanced 

by these scholars is that these technologies draw their notions of 
identity from the agendas of the institutions that implement them, 
and force subjects to accord with these notions. My own analysis 

takes this central critique as its point of departure. But, in contrast 

to the works discussed above, which examine the dominant social 

and political narratives that surround the development of biometric 

technology and its present application, I look at the empirical ma- 
terial of the technology and the technical processes themselves as a 
window into a cultural logic of visuality. This, in turn, may provide 

us with a source of critique and further insight into the limitations 

of this technology with regard to notions of identity and knowledge 

formation. 


Language and Visual Artifacts: Empirical Material 


As I have explained, previous cultural analyses of biometrics and 
AFR technology have focused on the contexts of their development 
and implementation. And while this is necessary in order to prob- 
lematize the use of these technologies, these studies tend to overlook 
the specificities of AFR technology. There has been little scholarship 
from within cultural studies and the humanities more broadly on 
the ways in which AFR actually performs recognition and how this 
may constitute and enculture a mode of perception. More often, the 
specificities of the technical processes of recognition are discussed 
and debated within the field of the computer and cognitive sciences, 
separated from any cultural investigation. When AFR technology 
has been discussed within cultural studies, it is for the most part 
treated as a monolithic method rather than as a variety of different 
methods that have evolved in specific ways. When the topic is scru- 
tinized in this way, individual AFR methods are largely ignored. In 
contrast to this approach, I choose to focus my own analysis on a 
specific AFR method. 
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My interest in examining the visuality of AFR has led me to fo- 
cus on a method that uses an eigenface approach to recognition, an 
approach that was developed in the early 1990s and is considered the 
first successful method in AFR technology. Many AFR systems are 
difficult to scrutinize because the technology is usually considered 
proprietary software.”° The eigenface approach, now considered a 
dated method, was open source from its inception. Because of its 
simplicity, it is often still used as a training tool by computer science 
students, so examples of its use are widely available on the internet. 
There are two primary reasons for my interest in this method. First- 
ly, the eigenface approach is considered a holistic method: that is, 
it takes into account the entire face, rather than isolated features 
of the face, and is, as such, designed to emulate human facilities of 
facial recognition. Secondly, as a part of its algorithmic processes 
of recognition, the eigenface approach was designed to produce an 
image, a visual artifact through which it is possible to enter into 
a visual analysis of its processes of recognition. The success of the 
eigenface approach to recognition made it a benchmark for the 
AFR methods that developed subsequently. And although it is now 
considered a somewhat dated method, its creation still remains an 
inflection point in the history of the development of AFR, shifting, 
for a moment, the direction of the technology toward holistic and 
pictorial processes of facial recognition. The eigenface approach is 
a simple method. In its simplicity, it reveals the technical processes 
that structure a successful AFR recognition operation. This basic 
structure can still be found in the more sophisticated AFR methods 
that have since been developed. 

My investigation into biometric facial recognition technology 
differs from the theoretical frameworks outlined in the previous 
section in that it analyzes AFR technology in relation to discourses 
of visuality and machinic vision. Nevertheless, this investigation 
does draw on the conclusions and some of the central conceptual- 
izations of previous cultural analyses. These conceptualizations act 


20 Lucas Introna and David Wood, “Picturing Algorithmic Surveillance: The 
Politics of Facial Recognition Systems,” Surveillance and Society 2, no. 2/3 
(2004): 185, https://doi.org/10.24908/ss.v2i2/3.3373. 


Chapter 1: Introduction 


as useful prompts, providing inroads into the analysis of the visu- 
ality ofan AFR recognition process. For example, drawing on Lyon’s 
aforementioned critique of biometrics and surveillance practices 
as enacting a form of “social sorting,” I ask what technical means 
AFR methods employ in this sorting process. How does this sorting 
equate to a form of vision? How are contemporary modes of percep- 
tion reduced to a sorting mechanism? As this line of questioning 
makes clear, this analysis approaches AFR technology by specifical- 
ly problematizing a visuality found in its processes, a visuality that 
has social and political implications. 

As I have mentioned, the eigenface approach is an earlier, “pic- 
torial” algorithmic method that reads the face holistically (rather 
than a feature extraction method that measures the distances be- 
tween features of the face). What sets the eigenface approach apart 
from other methods is that, as I mentioned above, it produces an 
image as a part of its algorithmic processes. The eigenface image 
and, importantly, the algorithmic recognition processes that pro- 
duce it, constitute the central departure point of this study. The 
production of this image was originally a way of providing the pro- 
grammer with a way to fix bugs in the algorithm. This image ap- 
pears as a phantom-like blur of multiple overlapped faces, which, to 
human eyes, lingers on the threshold of recognition. While the ei- 
genface image remains blurry to human vision, it contains a wealth 
of information that, for the processes of perception by the machine, 
equates to a form of clarity. The image provides a visual artifact 
that allows me to bring AFR processes into discussion with visual 
and cultural theory and discourses on machinic vision. I do this in 
two ways. Firstly, as I will explain in more depth in the theoretical 
section of the introduction, I present this image as an example of 
the changes in the role of the image as information brought about 
by the advent of digital networks. Secondly, I positioned the eigen- 
face image within a discourse of facial representation in the art 
historical genre of portraiture. Taken out of its original context of 
production, that is, as a tool for programmers, I understand the ei- 
genface image to be a machinic production of a portrait, depicting 
the human face from the perspective of machine. I investigate this 
in depth in the first part of this book, which relates the eigenface 
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image to a historical antecedent: the composite portrait. This line 
of analysis continues in the second part of the book, which looks at 
artistic interventions and discusses portraiture in the age of AFR 
technology. 

Another, related source of empirical material is provided by the 
scientific texts that describe the eigenface method. I was first intro- 
duced to the eigenface method by reviewing scientific journal arti- 
cles that outline the development of and experimentation with AFR 
methods by programmers and students in the field of computer sci- 
ences. The central texts for the analysis of the eigenface method are 
two articles written by its developers, Matthew Turk and Alex Pent- 
land. Aside from the articles written by Turk and Pentland, which 
record the initial findings of experiments using the eigenface meth- 
od, Turk later wrote another article, “Twenty Years of Eigenface,” 
which provides further insight into the impetus behind its original 
development and its relationship with other methods of AFR tech- 
nology.” I have also drawn on the work of the computer scientists 
Lawrence Sirovich and Michael Kirby, whose work preceded that 
of Turk and Pentland. Sirovich and Kirby applied the primary rep- 
resentational mechanism used in eigenface, Principal Component 
Analysis, to facial images, which influenced Turk and Pentland’s 
work.” These works are the primary scientific texts concerning the 
eigenface method. The method has become a popular algorithmic 
training tool for students of computer science, and there are nu- 
merous published scientific journal articles and blogs by computer 
science students and programmers that build on the method. These 
provide an endless source of information about eigenface images 
and a wealth of examples of such images. I have made use of some 
of these texts and images; because many of the blogs were written 
in such a way as to be accessible to laymen such as myself, these 
proved to be particularly useful. 
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A central challenge I confronted in seeking to understand the 
eigenface method through relevant scientific texts was that I had 
to understand specialist language from outside of my own field of 
expertise. Not only is there an abstraction of the image that occurs 
through the eigenface process; in analyzing the texts that outline 
this method, I also confronted an abstraction in the language used 
to describe it. These scientific journal articles communicate the 
method in four different forms: text, mathematical equations, pho- 
tographic and video images and graphs. In the course of my analy- 
sis, I found the images to be most relevant. The images correspond- 
ed to specific equations that explained more fully, for a specialized 
computer-science audience, the underlying procedures. Mathe- 
matical equations relating to the method and its processes figure 
prominently in the texts: for example, 27 times in the primary text 
by Turk and Pentland on the eigenface method.” It is important to 
note here that I do not include any mathematical equations in my 
own research; I lack knowledge of programming, and I am writ- 
ing a cultural analysis within a field of the humanities for an audi- 
ence within this discipline. Instead, I have focused on the language 
and the words chosen to describe the process and, of course, the 
eigenface images themselves. The captions written by the authors 
of the scientific articles, which at times take into account the aes- 
thetic aspect of the images, were also of importance for this study, 
for they express what was considered important in the image and 
supplement an explanation of the algorithmic process. Inevitably, 
the language as well as the images used by the developers became a 
central part of the empirical material. 

Ihave tried to bring to bear a cultural theoretical framework and 
concepts on a technical phenomenon that has predominantly been 
described in a specialist language and through mathematical equa- 
tions. I have thus faced a challenge in attempting to communicate 
the central mechanisms behind this process, as I understand them, 
without parroting the specialist language of the original sources 
and while also being careful not to oversimplify this explanation. 
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The language used in the humanities contrasts with that in the 
so-called “hard sciences,” such as the fields of applied mathemat- 
ics and computational sciences, specifically when it comes to the 
site of knowledge production. Scientific literature often describes 
phenomena from an assumed objective and distanced standpoint 
of observation, while, in contrast, as a humanities scholar, my own 
approach is explicitly interpretative, and it often requires an expla- 
nation of my own and others’ specific historical, cultural and social- 
ly situated vantage points. In short, in my observations, the site of 
knowledge production includes an “I,” and I freely admit my own 
limitations in confronting the specialist language used in these sci- 
entific texts, which make up a large part of the empirical material 
for this study.” 

I refer to in what follows include “The Scientific Development of 
Biometrics over the Last 40 Years,” by James L. Wayman, from The 
History of Information: A Comprehensive Handbook, and the recently 
declassified reports by computer scientist Woodrow Wilson Bled- 
soe, which concern some of the first attempts to develop AFR tech- 
nology, funded by the CIA.” These supplemental texts have helped 
me to explore further the broader contexts in which AFR technology 
has been used and developed. They also provide further informa- 
tion that situates the success of the eigenface method within the 
developmental trajectory of AFR technology. 

Another primary source of empirical material in this study is the 
collection of artists’ works explored in the second part of the book. 
The examples of artistic interventions included in this part bring 
out particular issues to do with the implementation of facial rec- 
ognition technology in society. Each artist relates the technology of 
facial recognition to specific discourses. Thomas Ruff’s study of the 
identification portrait as an artistic object creates a bridge between 
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AFR technology and a discourse on photographic portraiture. AFR 
technology still relies on photographs or video-still captures of the 
face and as such on a form of representation that connects back to 
the indexical relationship between identity and portraiture and the 
use of the photograph as a scientific document. Ruff’s work with 
the construction of faces in his series andere Portraits situates the 
composite portrait into the realm of art. As such, his work serves 
to connect the analysis of the eigenface method with the themes of 
the second part of the book. Zach Blas’s work brings AFR technology 
into a discussion with discourses of identity politics. His work con- 
fronts the gaze of AFR technology, and his sculptures materialize 
a contemporary strategy of activism through collectivism. Blas’s 
artwork highlights the social and political contexts in which AFR 
technology is implemented and the enmeshment of automated rec- 
ognition practices in the construction of contemporary identities. 
Trevor Paglen’s work is a study of visuality in light of discourses 
concerning the growing ubiquity and invisibility of algorithms in 
society. Paglen’s work not only disrupts how contemporary mean- 
ing is produced through algorithmic recognition processes but also 
explores the role of imagination. This collection of artworks ex- 
plores three essential aspects of the influence and role of contem- 
porary AFR technology: representation, identity and knowledge 
production. With the inclusion of these artworks, I hope to show 
that art has an important role to play as a source of theoretical re- 
flection on contemporary technologies, as well as offering a means 
of exposing their dangers and exploring the possible alternative fu- 
tures resulting from these technologies. 


Temporalities — History as Critical Inquiry: Methodology 


A primary method of historicization guides the critical analysis of 
the eigenface algorithm in the first part of this study. Eigenface is 
related to a historical antecedent, composite portraiture, which was 
invented by the anthropologist, statistician and founder of eugen- 
ics Francis Galton in the 1880s. These two methods are connected 
by the idea of merging vision with statistics in order to provide a 
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mechanism of recognition. The method of historicization as a tool 
for critically analyzing AFR technology relies on a concept pro- 
posed by African studies and surveillance scholar Simone Browne. 
Browne calls for a “critical biometric consciousness,” which includes 
ways of developing critical strategies that can reveal and allow us to 
scrutinize the technical development and socio-cultural implica- 
tions of biometric technologies.” Central to this call is an approach 
to biometric technologies that contrasts with its treatment as ob- 
jective and without history. This approach is directly opposed to the 
widely held belief that algorithms and biometric scanning exist as 
privileged, ahistorical practices of information technology that are 
objective and precise, a belief that validates the continued develop- 
ment and use of these technologies. Browne describes one method 
of developing a “critical biometric consciousness”: tracing histori- 
cal practices and antecedents that inform the social dynamics and 
technical development of contemporary biometric technologies.” 
In tracing the socio-historical lineage of the facial representation 
processes found in the eigenface method, this analysis answers 
Browne’s call. By linking the eigenface image with the historical an- 
tecedent of composite portraiture, we are able to reveal the cultural 
dynamics of facial representation that inform automated processes 
of recognition. In historicizing the eigenface image in this way, this 
study reveals not only a logic of recognition based on the merging 
of statistics with vision but also an embedded and situated cultur- 
al logic of facial representation within the eigenface’s algorithmic 
processes. 

The method of historicization in this analysis is inspired by re- 
cent scholarly work that has critically engaged with overarching, 
general terms by tracing the histories behind their meanings. Two 
key texts that have influenced this study are the book Objectivity, by 
Lorraine Daston and Peter Galison, and John Durham Peters’s work 
on the term information, specifically in his article “Information: 
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Notes Toward a Critical History.” Both of these works pursue a 
method of historicization, tracing the cultural meaning behind the 
practice-based terms “objectivity” and “information” as a means of 
demystifying their contemporary position as transparent and “om- 
nipresent idols”? and grounding their meaning in actual use and in 
modes of discourse. Like the scientific illustrations that appeared at 
the turn of the last century, recent technological developments and 
processes of information production are often, in the fields of cul- 
tural or aesthetic inquiry, treated as possessing a kind of objectivity. 
Often it is only the distance of history that allows us to engage in a 
more critical way with a scientific production that had been accept- 
ed as neutral in its own time. In tracing the historical continuities 
of the culturally embedded recognition practices found in an AFR 
method, I argue that we may contest the assumed scientific objec- 
tivity of the technological present. 

By relating the eigenface algorithm to a historical antecedent, 
the composite portrait, I put two sets of images into dialogue with 
one another: one the result of the representational mechanisms of 
the eigenface algorithm, and the other Galton’s historical, photo- 
graphic composite portraits. The relationship between these two 
sets of images involves a history of what I consider a way of seeing 
— a mode of perception that is based on statistical logic. In relating 
these two images, I am not trying to trace the origins of eigenface 
but rather relating it to a particular historical instance in order to 
reveal how its mode of perception is embedded within a specific 
cultural ethos. One thought that is central to my own approach to 
relating these two images comes from Walter Benjamin's “Theses 
on the Philosophy of History” (the last text he wrote before his pre- 
mature death). Benjamin writes: “The past can be seized only as an 
image which flashes up at the instant when it can be recognized 
and is never seen again [...] For every image of the past that is not 
recognized by the present as one of its own concerns threatens to 
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disappear irretrievably.” In my analysis, the composite portrait 
appears as a pictorial reference to a cultural ethos of social classi- 
fication from the past that, through contemporary practices of bio- 
metrics, finds a foothold in the present. The composite portrait is 
a contextually loaded image; embedded within it is an impulse to 
construct a social taxonomy, as part of Galton’s broader project of 
eugenics. This particular history is recalled by the classifying im- 
pulse of contemporary AFR technologies and is visualized through 
the eigenface image. In relating these two images, I do not mean 
to suggest that the historical relationship between them is self-ev- 
ident or that it exists on its own. Rather, to quote Benjamin again, 
“To articulate the past historically does not mean to recognize it ‘the 
way it really was’ (Ranke). It means to seize hold of a memory as it 
flashes up at a moment of danger.” It is in this spirit that I hold up 
the eigenface image and the composite portrait as objects of study 
connected by a relationship in which the past informs the present: 
the algorithmic processes used in the eigenface method are em- 
bedded with historical practices of racially charged classification, 
which makes clear the present danger posed by AFR technology and 
suggests a critique of its limitations and entanglements with regard 
to notions of recognition and identity. Tracing this historical link 
between the practice of composite portraiture and the representa- 
tional mechanism within the eigenface algorithm also reveals a link 
between discourses in photography — in its role in science and art 
- and the forms of representation that are found in current algorith- 
mic processes. 

The second part of this study draws on contemporary artists 
whose work bears on practices of facial recognition. The artists 
Thomas Ruff, Zach Blas and Trevor Paglen have each produced 
works that function within this study as further sources of theo- 
retical analysis, experimenting with and problematizing process- 
es of facial recognition. These artists were chosen because of their 
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explicit concern with some of the political and social implications 
of facial representation and recognition technologies. Their work 
supports an overarching narrative that runs through this analysis, 
a narrative that is concerned with the visuality and representation- 
al mechanisms of facial recognition. The artworks articulate the 
cultural contexts in which facial recognition technology intervenes, 
and they pose, in particular, the central criticism of these technol- 
ogies: that they reinforce normative categories of identity through 
reductive, technical procedures. I have chosen artworks composed 
using a variety of mediums in order to explore a range of responses 
to the implications of facial recognition. Ruff, Blas and Paglen work 
in the mediums of photography, sculpture and algorithmic-gen- 
erated prints, respectively. The artists’ works included within this 
study serve to supplement a central conceptual motif of the anal- 
ysis of eigenface (one artist, Paglen, directly references the meth- 
od in his work) through their engagement with and reformulation 
of composite forms of the face. Two of the artists, Blas and Paglen, 
have also produced written scholarly work alongside their artistic 
practices, and I have also made reference to this work. The concepts 
that these artists are working with are often articulated explicitly in 
these texts. I also had informal discussions with Blas and Paglen in 
person, which gave me further insight into the contexts and cultur- 
al circles in which they engage with their work. I conducted email 
interviews with Ruff, and I also refer to existing interviews with 
him that are included in catalogues of his work. All of the impres- 
sions I gained, both through written texts and in-person dialogue, 
provided further material for and insight into the approach they 
take towards facial recognition in their work. A primary motivation 
in choosing the artworks included in this study is that they use art 
as a vehicle for imagining alternatives to the technologies that in- 
creasingly govern our social life. 

The two parts of this study experiment with the separate tem- 
poral trajectories of past and future. In historicizing the eigenface 
method by relating it to the composite portrait, this analysis con- 
sciously counters a preemptive logic of the technology and, instead, 
points “backwards” toward the past as a source for informing the 
future. The first part of this study, which analyzes the eigenface 
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method, may be understood as referring primarily to the past, 
while the second part points toward the future through an explo- 
ration of contemporary artistic interventions and a discussion of 
possible alternative responses. 


Machinic Observer — Seeing as Recognition: 
Theoretical Framework 


Because this study focuses on the visuality of AFR technology, I 
relate the empirical material under discussion here to the work of 
contemporary scholars of the image who have sought to understand 
how visual sense perception may be understood as a machinic pro- 
cess. Most of this work relates to a theme in visual culture theory 
concerning the “disembodiment of the eye” that occurs through the 
use of visual technologies and is often analyzed through referenc- 
ing its expression in works of art. I argue that this theorization is 
useful in analyzing contemporary advanced visual technologies, 
but also that it has its limitations. Some of the ideas derived from 
this theory do not figure directly in the analysis, but I mention 
them here because they have inspired my thinking and my central 
approach to the analysis generally. 

Td like to begin an approach towards understanding machine 
vision with a reference to a scene from a film. In Charlie Chaplin’s 
Modern Times, the protagonist, played by Chaplin, is working in a 
factory and becomes enmeshed in the rhythm and movements 
of the industrial line.» His arm movements become one with the 
rhythm of the conveyer belt that brings him more and more objects 
to clonk. Later on, his entire body gets entangled in the workings of 
an enormous machine that appears like the insides of a clock, com- 
plete with springs, cogs and levers. Chaplin’s body, along with all its 
senses and movement, has been swallowed into the belly of the in- 
dustrial machine. The factory of Modern Times depicts an era of pro- 


32 “Chaplin Modern Times - Factory Scene (late afternoon),” YouTube video, 
4:13, posted by “Olaf V/s Minions,” September 5, 2015, https://www.you 
tube.com/watch?v=HPSK4zZtzLl. 
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duction in which the human body as a whole was employed within 
the industrial labor force of mass manufacturing, albeit in order to 
carry out specialized tasks. In contrast, thinking about how a per- 
son might become enmeshed in the labor and production processes 
of today’s Information Age, it is clear that the dominant part of our 
body that is forced to work is not the arm but rather the eyes. In- 
stead of the entire body becoming enmeshed in the rhythm of the 
industrial machine of production, the eye and the sense perception 
of sight become entangled within the operations of the informa- 
tion machine. The labor of looking has become a central demand 
within multiple contexts in the Information Age, where the chal- 
lenge is often in making things visible, and thereby known, within 
a sea of data. Widespread surveillance practices have made it that 
the amount of data produced now far outstrips human capacities 
of understanding. Algorithms have been developed to codify this 
visual labor, to inspect and sort through the mass of data, to make 
“sense” through the automation of a form of sense perception. In 
this scenario, vision as a form of sense perception connected to the 
production of knowledge and meaning has become aligned with the 
logic, operations and rhythm (or temporalities) of the machine. 

It is important to begin with a concept of machine vision, as it 
is a central concept that underlies my whole approach to analyzing 
AFR technology, as being a contemporary example. I am interested 
in AFR technology as an object of study because of the way it cod- 
ifies and automates visual sense perception within an operation of 
recognition. Scholars have theorized machine vision within a dis- 
course of art history and visual culture. John Johnston has outlined 
a concept of “machinic vision,” which he defines as “not only an 
environment of interacting machine and human-machine systems 
but a field of decoded perceptions that, whether or not produced by 
or issuing from these machines, assume their full intelligibility only 
in relation to them.”” In his use of the term “machinic,” Johnston is 
drawing on the work of Gilles Deleuze and Felix Guattari and their 
understanding of the term as suggesting an “assemblage,” that is, 


33 John Johnston, “Machinic Vision,” Critical Inquiry 26, no. 1 (Autumn 1999): 
27, https://doi.org/10.1086/448951. 
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“a type of working relationship among heterogeneous elements.”™ 
Deleuze distinguishes this understanding of the machine, as a 
binary opposite of the human, or the organic. Johnston describes 
“machinic vision” as a disembodied perspective that derives from an 
assemblage of machine and human practices. He references expres- 
sions ofa machinic vision, for example, through art in the paintings 
of Francis Bacon and in cinema through the work of Dziga Vertov. 
Again drawing on Deleuze, Johnston describes a machinic vision as 
rejecting a centered world view and a “phenomenological point of 
departure.”® Instead, Johnston says that machinic vision is a “pure 
vision of a non-human eye, of an eye which would be in things.” In 
this way, the act of seeing is liberated from an anchored point and 
becomes mobile — it may be found in objects, take on a molecular 
form and operate beyond human scale. 

Two key movements of Deleuze’s definition of the “machinic” 
that Johnston applies to vision are deterritorialization and reter- 
ritorialization. Johnston describes that the former occurs when a 
form of visual perception is freed from the person that is doing the 
seeing, and the latter is when that seeing is “recoded,” that is, re- 
contextualized and expressed in new form and, as a result, produc- 
es new meaning. Johnston argues that what must be understood is 
this recoding. He outlines the problem of approaching these mo- 
ments and expressions as they occur in forms other than the tradi- 
tional forms of art he previously mentioned, particularly when they 
occur in digital images. He states that: 


in order for there to be a deterritorialization and thus a decoding of 
perception, there must be a movement toward the outside of an as- 
semblage and beyond its coding apparatus, a movement carrying 
us into a zone where images become indiscernible, often asa result 
of particle-ization of elements [...] But for the digital image there is 
no outside, only the vast telecommunications networks that sup- 
port it and in which it is instantiated as data. Instead of an outside, 


34 Ibid., 28. 
35 Ibid., 34. 
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the digital image seems only to have an electronic underside [...] 
which cannot be rendered visible. 


Johnston outlines the challenges of a circulatory and self—referen- 
tial system found with images within digital networks. In the fol- 
lowing analysis, I address the problem of revealing an “underside” 
of the digital image by basing my inquiry on a visual artefact of an 
AFR process. Specifically, I will approach the challenges posed by 
digital networks through a discussion on the recontextualization 
of the algorithmically produced eigenface image and through the 
examination of the work of artists who have relied on non-digital 
mediums. The movements that Johnston define are key in this anal- 
ysis to understanding how an AFR process deterritorialize visual 
perception in an algorithmic process and reterritorialize it in an 
operation of recognition. I specifically problematize the “recoding” 
of vision through the eigenface algorithm and its ability to produce 
new meaning in terms of the identity of the subject being recog- 
nized. 

Paul Virilio is someone who figures centrally in the theorization 
of machine vision and although his conceptualization on the topic 
does not appear in this analysis, he has nonetheless inspired my ap- 
proach and thinking and therefore needs to be mentioned here. His 
book The Vision Machine predicted the widespread implementation 
of machine vision in society. He writes, “Unless you are Lewis Car- 
roll, it is hard to imagine the viewpoint of a doorknob or a button 
on a cardigan. Unless you are Paul Klee, it is not easy to imagine 
artificial contemplation, the wide-awake dream of a population of 
objects all busy staring at you.”” In his brief and singular reference 
to an “artificial contemplation,” Virilio refers to works of literature 
and art production as mediums through which one can imagine and 
communicate the life of objects and in particular the perspective 
from which objects can perceive a subject. My own methodology 
is inspired by Virilio’s suggestion that mediums outside of techni- 


36 Ibid., 39. 


37 Paul Virilio, The Vision Machine, trans. Julie Rose (Bloomington, IN: India- 
na University Press, 1994), 137. 
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cal research might allow one to make sense of “artificial contem- 
plation.” Given the ways in which an automated process of vision 
considers and makes meaning of physical phenomena, this seems 
a promising suggestion. This analysis considers Virilio’s suggestion 
in relation to algorithmic processes, and it attempts to formulate 
a narrative around algorithmic contemplation as it relates to AFR 
technology. As such, I like to bring Browne’s aforementioned argu- 
ment for a “critical biometric consciousness” into dialogue with Vi- 
rilio: a kind of discourse between a social consciousness confronted 
with this algorithmic contemplation, whereby, at this intersection, it 
may map a critical terrain on the ways in which algorithms can give 
accounts to themselves through the perspective of an automated 
sense perception. 

The artist Harun Farocki is another figure whose conceptualiza- 
tion figures in this analysis in particular his work with machine vi- 
sion. Farocki’s concept of “operational images” has been influential 
in understanding the ecology of images that result from the output 
of machine vision operations. He defines the term as “images that 
do not represent an object, but rather are part of an operation.”** Fa- 
rocki has described the operational image as an image that is unlike 
traditional images in that it is made neither to entertain nor to sell; 
operational images, he says, “are information and not really imag- 
es.” He further describes the operational image as implementing 
a new visual regime where images are made by machines for other 
machines and, “the aesthetics of which are not intended.™° The op- 
erational image poses a shift in the role of images from represen- 
tational to operational, and as such ascribes to images an agency 
within a prescribed function. This is particularly pertinent to both 
parts of my analysis, but especially to the analysis of eigenface, 
which begins with the algorithmically produced eigenface image. 
The eigenface image is an example of an operational image, for it is 


38 Harun Farocki, “Phantom Images,” trans. Brian Poole, Public 29 (Spring 
2004): 17 [12-22]. 

39 Harun Farocki, War at a Distance, video (Berlin: Harun Farocki Filmpro- 
duktion, ZDF, 2003). 
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generated only through its recognitive operation. Yet, the eigenface 
image overlaps the two registers of representation and operation in 
that it harbors both. The eigenface image is part of the operation of 
recognition in machine vision yet it is also an image that depicts the 
representation of faces (or the representational mechanism) within 
an algorithmic process. 

Another aspect in framing the theoretical approach of this anal- 
ysis, is relating an understanding of machine vision within a dis- 
course on a historical construction of vision. Jonathan Crary’s book 
Techniques of the Observer asks pertinent questions and provides cer- 
tain points of departure that have contributed to the development 
of this outlook. Crary attends to the phenomenon of the observer 
(rather than to the artistic image as a source) in investigating vision 
as a historical construct. He states: 


For the problem of the observer is the field on which vision in his- 
tory can be said to materialise, to become itself visible. Vision 
and its effects are always inseparable from the possibilities of an 
observing subject who is both the historical product and the site 
of certain practices, techniques, institutions and procedures of 
subjectification.”| 


Attending to the etymology of the term “observer” — meaning “to 
conform one’s action and to comply with” - Crary describes the ob- 
server as “one who sees within a prescribed set of possibilities, one 
who is embedded in a system of conventions and limitations.” The 
materialization of vision, making vision, as a site of practices and 
techniques, “visible,” is one of the primary concerns of this study. 
My concern is revealing the contemporary machinic observer as a 
primary observing subject. Joining Johnston’s understanding of the 
term “machinic” as an assemblage with Crary’s “observer,” we can 
begin to explore these concepts as they manifest themselves in our 


41 Jonathan Crary, Techniques of the Observer: On Vision and Modernity in the 
Nineteenth Century (Cambridge, MA: MIT Press, 1990), 5 (Italics in origi- 
nal). 

42 Ibid., 6. 
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contemporary world, in the empirical material of AFR technology. 
In pursuing this concern, it is possible to reveal, in contemporary 
machine vision practices, certain historical continuities and dis- 
continuities in the organization of vision. I seek to understand the 
materialization of the machinic observer’s “system of conventions 
and limitations.” Crary’s focus on the organization and construc- 
tion of vision as it is materialized through the human observer an- 
ticipates an organization of vision that is fully realized through the 
automated processes of AFR technology. His study reveals how vi- 
sion has been constructed historically through the organizations of 
technology, society and science. 

The theorists mentioned here so far have all contributed to a 
theoretical approach in how to understand forms of seeing that are 
machinic — as an assemblage, as codified, as industrialized and 
historically constructed. This provides a broader context within 
which to approach AFR as a technology in which the sense percep- 
tion of vision gets enveloped by an automated process of recogni- 
tion. In understanding AFR technology as one instance of a machin- 
ic observer, we can begin to formulate an account of how vision can 
be codified in a specific operation and within the conventions of 
“recognition.” Yet, as I have explained, this codification is not with- 
out a history; it is, rather, embedded with historical and cultural 
practices of seeing. 

In what follows, I use the phrase “a way of seeing” when refer- 
ring to AFR technology as a way of making clear that these process- 
es are embedded in cultural and historical contexts. This phrase is 
a direct reference to the work of John Berger, who seeks to under- 
stand the situated perspective of the observer in relation to works 
in art history. He mentions an especially central dynamic (albeit in 
a different context) between seeing and knowledge, which I argue 
underlies a central dynamic of machinic vision. He begins his book 
Ways of Seeing with the simple yet bold statement: “Seeing comes be- 
fore words [...] The relation between what we see and what we know 
is never settled.” He describes a gap between knowing something 
and seeing it. This gap is also a space in which a fluidity of mean- 


43 John Berger, Ways of Seeing (London: Penguin Books, 1972), 7. 
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ing can exist — a fluidity of meaning that occurs before it can be 
cemented in the meeting of the two. As an example, Berger refers 
to the artist Henri Magritte, whose paintings conjure up this gap as 
dissonance, in the play between text and form. Seeing as recogni- 
tion through machine vision, reverses this order whereby knowing 
comes before seeing. An algorithm must know a face before it can 
recognize it. Further, an image recognition algorithm must “know” 
the object before it can successfully perform an operation of recog- 
nition. This seemingly simple understanding of the order of things 
in vision prompts an important theoretical critique in the form of a 
question, one that underlies the analysis set out here: if a recogni- 
tion algorithm must know the object before it can recognize it, how 
does that transform and limit its knowledge of the things it sees? 

In addition to the theoretical framework concerning machine 
vision, this analysis is also informed and framed by an account of 
the body and of the imaging of the body as well as theories of bio- 
politics. I have drawn on the work of Allen Sekula and his study of 
the intersection of the body and the archive. In addition, the the- 
ory of biopolitics, specifically what it has to say about the body in 
relation to information and being made “intelligible,” has also in- 
formed my understanding of the shift in knowledge production 
that occurs through the introduction of AFR technologies. As I have 
already established, biometric systems, as risk mitigation and se- 
curity technologies, have specific desired outcomes, namely, to es- 
tablish identity as clearly as possible in a geopolitical landscape of 
uncertainty and in the context of border control. The application of 
biometrics in these arenas submits the body to processes in which 
it gets translated into data, in the service of institutional ends. This 
makes it possible to inscribe the body with the institutional status- 
es of nationality and other terms and identities constructed outside 
the person’s subjective experience of his or her body. AFR technol- 
ogies, and biometrics in general, operate by both negating and le- 
gitimizing identity. What is sacrificed within this paradigm is the 
subjective experience of the body; one’s own construction of self, 
grounded in the human sensorium, gets negated when confronted 
with an identity imposed from the outside. The identity and status 
associated with the biometric identity then has the operative func- 
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tion of dividing an individual not only from other bodies but also 
from herself. Biometrics creates a situation where one’s body can 
act as a witness against oneself. It involves a process of dividing the 
surface of the individual, if you will, from what is beneath the skin, 
in which the body, in becoming a docile body subjugated to gover- 
nance, becomes a subject of the state through its transformation 
into an image of the self. 

This process of subjecting the human body to political inscrip- 
tion recalls Michel Foucault and what he terms “dividing practices,” 
in which an individual's identity becomes defined in relation to so- 
ciety, and becomes vulnerable to exclusion according to prescribed 
categories, such as the “mad” and the “sane.” This division between 
the internal subjective experience of selfand the external naming of 
an institutional identity becomes part of the mechanisms through 
which an individual is turned into a subject of governance. Fou- 
cault also describes two registers through which the subject be- 
comes transformed into a “docile body” under state power and is 
both subject and subjugated to governance.* The first register is the 
“anatomico-metaphysical body” or “intelligible” body, that is, the 
body as an object of knowledge, for example the body as the object 
of certain forms of measurement.“ The second is the “technico-po- 
litical” or “useful” body that submits to and is used in regimes of 
discipline and correction in institutions such as the army, hospital, 
prison and school. Through the practice of biometrics, the scanned 
body inhabits both of these registers: biometrics provides a body 
that can be read and a body that is of use in virtue of its ability to 
produce information. This accords well with what Foucault says 
of dividing practices: when one is categorized by social and state 
norms, one experiences a divide either inside oneself or between 
oneself and others. Biometric practices, again, encompass both of 


44 Michel Foucault, “The Subject and Power,” Critical Inquiry 8, no. 4 (Sum- 
mer, 1982):777-778. 

45 Foucault references the work of Julien Offray de la Mettrie, Lhomme ma- 
chine, 1747. 

46 Michel Foucault, Discipline and Punish (London: Penguin Books, 1991), 
136. 
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these registers, in that, where the body can act as witness against 
oneself in the service of the categorical identities and statuses of the 
state, there exists a divide inside oneself, dividing what is beneath 
the skin from what is on the surface of it. One’s embodied experi- 
ence of subjectivity and the self is transformed into an image and 
becomes subjected to the state. The identity and status associated 
with the biometric identity then function to divide the individual 
from others. 

How a body can be read and of use through a practice of biomet- 
rics is further referenced in the analysis through the work of Irma 
van der Ploeg, in particular her analyses on the “informatization 
of the body.” Van der Ploeg argues that biometrics sets forth a new 
body ontology and she discusses biometric practices in relation to 
the supposed dichotomy between information - as in the “Informa- 
tion Age” — and materiality. She explains that technologies such as 
biometrics, which blur the line between “bodily matter” and “bodily 
information,” have transformative effects on the level of embodi- 
ment and that this is important because “embodiment is central to 
individuality and identity in a way that my social security number 
or car rental records are not.” Van der Ploeg provides an import- 
ant discussion of how biometrics traverses multiple boundaries of 
knowledge and simultaneously demands new definitions of bodily 
integrity. Her arguments provide further support for the claim that 
the implementation of biometric processes transform our under- 
standing of the subject. 


Outline 


This dissertation is divided into two main parts. The first part is a 
close analysis of the AFR method of eigenface and a historicization 
of the processes involved with this method. The second part at- 


47 Irma van der Ploeg, “Biometrics and the Body as Information: Normative 
Issues of the Socio-technical Coding of the Body,” in Surveillance as Social 
Sorting: Privacy, Risk and Digital Discrimination, ed. David Lyon (London: 
Routledge, 2003), 70. 
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tempts to move toward a reimagining of facial recognition technol- 
ogy and its technical processes through an examination of the work 
of contemporary artists whose pieces reveal possible alternatives to 
the logic of this technology. Taken together, these two parts con- 
stitute an analysis of the visuality of automated facial recognition. 
The first part begins with an introductory chapter on AFR tech- 
nology, briefly outlining the problems and challenges that have 
arisen in the course of its development. The discussion then moves 
to an overview of the eigenface method and its technical features, 
outlining three aspects of its recognition process: Principal Com- 
ponent Analysis (PCA), the eigenvector and the “face space.” The 
eigenface method is then related to the historical practice of com- 
posite portraiture as developed by Francis Galton, and I discuss 
the use of statistics as a way of seeing. Galton’s practice of finding 
statistically salient patterns in composite portraits in order to cre- 
ate images of “types” serves to open up an inquiry into reductive 
forms of knowledge production. Ludwig Wittgenstein, who him- 
self produced a composite portrait inspired by Galton, is brought 
into this discussion as the source of an alternative approach to the 
composite that does not follow the statistical logic of perception 
put forward by Galton. Wittgenstein refers to the composite por- 
trait in the course of his philosophical investigations of language; 
for Wittgenstein, the composite portrait is both a rendering of the 
overlap of similarities between forms and a depiction of the partic- 
ularities, and it thus resists any theory of generality. He refers to the 
composite specifically in relation to his idea of family resemblance 
and his account of concept formation. As I show, his perceptual 
interest in the composite lies not in the static center of the image, 
where the average can be perceived, but rather in the outlying ar- 
eas, where a perceptual movement between forms can be perceived. 
This approach is discussed further with reference to Wittgenstein’s 
concepts of “aspect perception” and “aspect blindness.” Wittgen- 
stein presents an alternative aesthetic approach to the composite 
portrait, one that directly inverts Galton’s approach and, by impli- 
cation, the statistical logic that underlies the eigenface algorithm. 
In this, Wittgenstein opens up a perceptual space in the composite 
portrait that defies the singular output of recognition, and he in- 
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stead argues for a perceptual clarity grounded in being able to see 
the fluidity of forms. I argue that, in relation to the contemporary 
context of ubiquitous facial recognition technology, this Wittgen- 
steinian approach has both utopian and political implications. 

The second part of the book focuses on artistic interventions. The 
artworks included in this study are understood as providing visual- 
izations of the algorithmic process as well as of critical theoretical 
concepts. The work of artists provides further context to the appli- 
cation of contemporary facial recognition technologies, while also 
reconceptualizing and reimagining the use of such technologies 
in society. Working specifically through a visual vocabulary, artists 
are in a unique position to articulate how these technologies play a 
primary role in enculturing processes of perception and representa- 
tion. 

The second part draws on the analysis provided in the first. It be- 
gins with a survey of composite portraits in art, before introducing 
the work of Thomas Ruff, in particular his studies of identification 
portraiture and composite portraits in his series andere Portraits 
(1995). This examination of Ruff’s andere Portraits looks into his ap- 
propriation of the Minolta Montage Unit, a photographic apparatus 
first utilized by police in Berlin in the 1970s to produce composite 
mugshots. Although Ruff’s series does not directly deal with an 
algorithmic process, his work is relevant to my discussion because 
it problematizes the forms of representation in facial recognition 
practices. By taking portraits of portraits, he raises questions about 
the forms of representation found in identification photography 
and confronts the diffused relationships between archival practic- 
es and the production of subjects, citizenship and identity. As such, 
his work expresses some themes that are latent in the eigenface im- 
age, and it can inform our understanding of the structures of repre- 
sentation found in facial recognition systems. I also argue that his 
work may be understood as a contextualization and visualization 
of Wittgenstein’s concept of aspect perception. I analyze individu- 
al images from the andere Portraits series in order to discuss Ruff’s 
rendering of the fluidity of gender, which serves as a critique of a 
binary understanding of identity. 
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The next chapter covers the work of Zach Blas, and particularly 
his piece Facial Weaponization Suite. With this, the discussion of ar- 
tistic interventions moves into the more contemporary context of 
facial recognition algorithms, presenting a conceptual strategy of 
resistance to the algorithmic gaze. I discuss Blas’s own theoretical 
writings, which complement his art production with references to 
Edouard Glissant’s concept of “opacity.” Blas advocates a right to a 
specifically “informational” opacity, that is, a right to not be seen, 
identified and archived. Blas’s work may be understood as a sculp- 
tural form of the composite portrait, that is utilized as a mask in 
order to resist identification. Blas’s work confronts the reductive 
practices of automated recognition through a strategy using an 
excess of facial forms. In this way, I argue, Blas’s work inverts the 
algorithm’s representational mechanisms to resist an operation of 
algorithmic recognition. 

The third chapter in this part is a study of the work of Trevor 
Paglen. Paglen’s artistic approach can be understood in terms of 
a notion of transparency. Rather than confronting the end gaze 
of facial recognition technologies, Paglen’s work ventures into the 
processes of algorithmic recognition that underlie this gaze, tak- 
ing, as its artistic material, the training sets used by image recog- 
nition algorithms. In this way, I argue Paglen utilizes the algorith- 
mic process as a ready-made object that has become ubiquitous in 
contemporary society and modifies its output in order to produce 
artistic objects. My study of Paglen’s work focuses on two of his 
pieces: Adversarially Evolved Hallucination (2017) and Eigenface (Even 
the Dead are Not Safe) (2017). In the latter, through the subjects that 
Paglen chooses for his portraits, the eigenface image is related both 
to discourses in contemporary art and to discussions in political 
philosophy. These two works are further explored in relation to the 
goal of reimagining the very structures and outcomes that define 
algorithmic perception and its expanding role in the production of 
knowledge in society. 

The concluding chapter summarizes the main findings and rec- 
ommendations of this study and highlights potential areas for fur- 
ther research. This chapter also discusses the overarching theme of 
the composite as an aesthetic form that structures knowledge pro- 
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duction in the Information Age. In closing, I offer some thoughts 
about controversial uses of facial recognition technology that came 
to light during the final stages of writing this book. 


Part l: Aesthetics of an Algorithm 


Chapter 2: Eigenface 


Background: Eigenface in the Development 
of AFR Technology 


Cultural historian Wolfgang Schivelbusch reminds us that, “as 
technological methods recede in importance, they reappear as 
an object of study.”! It has been almost thirty years since the AFR 
method known as eigenface was first developed. Since its intro- 
duction, more sophisticated methods of AFR have been developed 
and have become more widely used. Yet the eigenface method re- 
mains important to consider when investigating the visuality of 
AFR. The eigenface method served as a basis for the development 
of other procedures that went on to expand the possibilities of AFR. 
Its success validated the use of facial recognition algorithms and 
spurred on the development of facial recognition technology. For 
example, the development of the eigenface method made possible 
the introduction of fisherfaces, a more refined method allowing 
for a more precise recognition process. The methods of facial rec- 
ognition more often used today rely on deep neural networks and 
other feature-based methods. The Viola-Jones algorithm (2001), for 
instance, uses a method of Haar Cascade to detect objects through 
superimposition, training an algorithm to differentiate true pos- 
itives from increasing variations of false negatives. The methods 
that have since developed do not produce an image or deploy an ab- 


1 Wolfgang Schivelbusch, The Railway Journey: The Industrialization of Time 
and Space in the 19th Century (Berkeley, CA: The University of California 
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stract notion of “seeing” through an algorithmic process. Program- 
mers argue that today’s AFR algorithms do not remotely resemble 
the human capacity of vision. 

The eigenface method was introduced after what has been 
termed the “AI Winter” of the 1980s, when pessimism surrounded 
the technology development sector and funding for new technolo- 
gy declined. Nevertheless, this decade also saw the first use of the 
term “biometrics” in the media and in public forums to describe 
automated recognition systems. This was the result of a growing 
awareness in the field of automated recognition technologies” that 
led to a “Coming of Age” of technological development in the 1970s. 
In the 1990s, the decade in which eigenface was introduced, the first 
“Biometric Consortium” was held, organized by the research divi- 
sion of the US National Security Agency (NSA).? 

The eigenface method was first developed as a fully automated 
biometric facial recognition system by two MIT scientists, Matthew 
Turk and Alex Pentland, who developed it in conjunction with Arbi- 
tron,‘ a television ratings company, for the purpose of monitoring 
ratings. Working within this consumer-marketing context, their 
goal was, as they have stated, to “develop a computational model 
of face recognition that is fast, reasonably simple, and accurate 
in constrained environments such as an office or a household.” It 
was designed to be used in TV sets to determine which individuals 
within a household were watching TV at which times, feeding this 
information into consumer ratings for specific television programs. 
Essentially, eigenface was designed to be integrated into a kind of 
TV that watches you as you watch it. Given this context, the ide- 
al situation for the algorithm to operate in is a real-time situation 
in which people are sedentary. Ideally, the faces to be recognized 
would all be positioned conveniently, that is, squarely in front of the 
TV screen, in a neutral, forward-facing pose. This pose is familiar 
from identification photographs, so it was not a great leap to imag- 


2 Wayman, “Scientific Development of Biometrics,” 266. 
3 Ibid., 269. 
4 Turk, “Over Twenty Years of Eigenface,” 2. 


5 Turk and Pentland, “Eigenfaces for Recognition,” 71. 
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ine that eigenface might potentially have uses outside of the context 
of TV ratings, particularly in areas involving the use of identifica- 
tion documents. The multi-faceted application of facial recognition 
methods can be understood as a bleeding through from the sphere 
of consumer interests to the socio-political arenas of risk and con- 
trol. 

At the time of its introduction in 1991, the eigenface method was 
considered one of the first facial recognition methods successfully 
to perform face detection and recognition in real time. Before ei- 
genface, automated recognition methods had focused on “feature 
extraction,” that is, the recognition of isolated features (also termed 

“landmarks”) of the face, such as the eyes, nose and mouth, and the 

measurement of the distances between these features. The eigen- 
face method departs from this earlier approach by relying on a rep- 
resentational mechanism that takes into account a holistic repre- 
sentation of the face rather than its isolated features. In doing so, 
the eigenface algorithm had a built-in capacity to detect faces, as 
well as to locate, track and classify a subject’s face. 


Bledsoe: “The Model Method in Facial Recognition” 


The introduction of the eigenface method marked a shift in the de- 
velopment of AFR technology not only because its seemingly sim- 
ple technique was successful but also because it performed recog- 
nition differently from the AFR systems that came before it (and 
from those that would come after). The eigenface method shifted 
the approach of AFR methods towards a holistic representation of 
the human face. To understand why eigenface was considered suc- 
cessful, it helps to understand the original problems and challeng- 
es to which this technology was developed as a response. The first 
attempts to codify and automate facial recognition in an operable 
process were documented in reports authored by the computer sci- 
entist Woodrow Wilson Bledsoe, considered one of the founders of 
artificial intelligence. Two of these reports were only recently made 
publicly available (in 2014). They had previously been classified, 
while references to these reports described them as being commis- 
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sioned by an “unnamed intelligence agency.” Alongside the recent 
discovery and publication of these reports, it has also emerged that 
they were funded by a CIA front organization, the King-Hurley Re- 
search Group.’ 


Figure 1: “Examples of photograph pairs used in the study,” 
Woodrow W. Bledsoe, 1964. 


6 Michael Ballantyne, Robert S. Boyer, and Larry Hines, “Woody Bledsoe: 
His Life and Legacy,” AI Magazine 17, no. 1 (Spring 1996): 7-20, https://doi. 
org/10.1609/aimag.v17i1.1207 and Wayman, “Scientific Development of Bio- 
metrics,” 264. 

7 The two Bledsoe reports were made publicly available thanks to the efforts 
of researcher Justin Lange, who, in 2013, as a master’s student at the Inter- 
active Telecommunications Program at New York University, was able to 
successfully retrieve them from the Dolph Briscoe Center for American His- 
tory at the University of Texas. On the basis of his research into Bledsoe’s re- 
ports, Lange concludes that the King Hurley Research Group, whose name 
is included in the title of one of these reports, was a front organization for 
the CIA. Lange’s claim is corroborated by Christopher Robbins’s book The 
Invisible Air Force: The Story of the CIA’s Secret Airline (London: Macmillan, 
1981). Thank you to artist Kyle McDonald for information about this back- 
story and for connecting me with Justin Lange. 
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Thus, AFR technology seems to have originated as a mechanism for 
surveillance and intelligence accumulation in the context of nation- 
al security operations, specifically under auspices of the research 
arm of the CIA. 


Figure 2: “Double exposure shows that the two different subjects are 


surprisingly similar on a point by point basis,” Woodrow W. Bledsoe, 
1964. 


k 


Figure 3: “Double exposure shows that the two poses of the same subject 
have very little in common when considered on a point by point basis,” 
Woodrow W. Bledsoe, 1964. 


There are many fascinating details in these original reports of the 
pioneering and systematic attempt to automate the recognition 
of the face, but the most relevant to this discussion is Bledsoe’s 
account of the problems and challenges of the task and how he 
chooses to visualize these. In Bledsoe’s report, “The Model Meth- 
od in Facial Recognition,” Bledsoe notes that “one of the most chal- 
lenging areas of pattern recognition is the identification of human 
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photographs by machine.”* He goes on to describe the difficulties of 
recognition on the basis of facial photographs, given the variations 
in age, expression, the angle and rotation of the face and in the di- 
rection and intensity of light hitting the face. This original account 
of the challenges faced by automated recognition is still mentioned 
in scientific papers published on AFR technology today. As part of 
this report, Bledsoe included a collection of training images, a data- 
set that he drew upon (figure 1). It is still not known where these 
original training images originated from, but they are all of white 
males of various ages.” 

Bledsoe uses the method of double exposure to visualize some 
of these challenges confronting automated recognition. He pairs 
two portraits together that differ in the direction of facial rotation 
and superimposing one over the other to make clear the differenc- 
es between a reference image and a capture image. Figure 2 is an 
example of this. It presents the facial faces of two men of different 
ages, yet these images exhibit the same direction of lighting and the 
same head rotation. Once superimposed, the faces merge. Accord- 
ing to Bledsoe’s caption, “two different subjects are surprisingly 
similar on a point-by-point basis.” In a contrasting series of imag- 
es, Bledsoe presents two images of the same man, now with differ- 
ent lighting direction and head rotation (figure 3). Here, he presents 
a double-exposure image depicting a jumble of ears, eyes and hair. 
Bledsoe describes how the two images of the same subject have very 
little in common. Bledsoe’s exercise in superimposed photographic 
depiction conveys the central problem of similarity and difference 
in AFR technology, and thus the potential for false positives and 
false negatives. Bledsoe’s early visual experiments with the over- 
laying of facial images foreshadow the eventual solution found in 
the eigenface method. The method of statistical pattern recognition 


8 Woodrow Wilson Bledsoe, “Ihe Model Method in Facial Recognition,” 
Technical Report PRI 15 (Palo Alto, CA: Panoramic Research, Inc., 1964), 2. 

9 Based on the demographics, Lange speculates that these portraits are from 
a criminal database. Bledsoe himself cites the work of Alphonse Bertillon 
and Cesare Lombroso on the “criminal man.” 

10 Bledsoe “The Model Method,” 7. 
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used in eigenface essentially utilizes these differences and similar- 
ities and encodes them. This early production of a binary composite 
reveals the kinds of challenges faced in the initial stages of the de- 
velopment of AFR technology. Yet Bledsoe’s composites also antic- 
ipate the eigenface approach, which utilizes the representation of 
similarity and difference through superimposition and transforms 
this into a mechanism of successful recognition. 


Representational Mechanisms and the Machinic Observer 


The inspiration for the eigenface algorithm was the thought that it 
might be possible for algorithmic processes to mimic the processes 
of human recognition. This was expressed through a shift in the al- 
gorithmic modeling of the face from a concentration on its isolated 
features to a holistic representation of the face. Turk and Pentland 
were working at the intersection between physiology, information 
theory and the psychology of face recognition. They argued that 
human face recognition does not occur through the perception of 
individual facial features and the relationships between them, as 
previous algorithmic research had suggested. They stated: “individ- 
ual features and their immediate relationships comprise an insuffi- 
cient representation to account for the performance of adult human 
face identification.” Instead, Turk and Pentland set out to build an 
algorithmic facial recognition technique that could produce a ho- 
listic representation of the face. This shift is important to consider. 
Algorithms depend on some form of representation and reproduc- 
tion of the face. The form and design of this representation in the 
algorithm provides a foundation that allows the AFR technology to 
learn to recognize a human face. Marcin Mitkowski has defined the 
representational mechanism in computational learning processes 
as having three capacities: “it can refer to a target of representation, 
it can identify information about the target that is relevant for its 
own interests and goals, and it can evaluate the value of the infor- 


11 Turk and Pentland, “Eigenfaces for Recognition,” 72. 
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mation based on environmental feedback.” Here I use the term 
“representational mechanism” to refer to a part of the AFR process 
that is responsible for identifying information relating to a face that 
is relevant to a process of recognition. Eigenface relies on the use 
of a statistical method of pattern recognition as a representational 
mechanism. It is this representational mechanism which is depict- 
ed in the eigenface image. 


Figure 4: “Sample face on top and its caricature below it.” 
Sirovich, Kirby, 1987. 


In order to produce the primary representational mechanism for 
eigenface, Turk and Pentland drew on the work of two scholars of 
applied mathematics at Brown University, Lawrence Sirovich and 


12 Marcin Mitkowski, Explaining the Computational Mind (Cambridge, MA: 
MIT Press, 2013), 156. 
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Michael Kirby. In 1987, Sirovich and Kirby published a paper on the 
use of a statistical method of pattern recognition called Principal 
Component Analysis (PCA), which they applied to facial images in 
order to produce what they called “eigenpictures.”" In their paper, 
Sirovich and Kirby speculate about how humans perform the func- 
tion of recognizing faces and note how adept humans are at this 
complex task: humans can recognize an almost infinite number of 
different faces. They propose that humans are able to recognize so 
many faces because we engage in a process of deduction in relation 
to facial characteristics; that is, we recognize the characteristics of a 
face that depart from a kind of characteristic mean. They addition- 
ally propose a mathematical translation of this process, applied to 
multiple facial images, as a possible model for how humans recog- 
nize faces. For example, at one point they refer in the text to pho- 
tographs of two faces (figure 4), with one photograph having had 
this mathematical translation of the deduction process (referred to 
as the Fourier method) applied to it, a photo they refer to as a “car- 
icature,” and the other without having had this reduction applied 
to it. The photographs, they say, appear “virtually the same to us.” 
The original (on top) is a still from a video, whereas the other, having 
undergone a mathematical reduction of pattern recognition, is its 
transformation into a computational image. When looking at these 
two facial images, I see a significant difference between the “carica- 
ture” and the original; the caricature looks as though it has under- 
gone what is referred to in photography as a solarization process, 
whereby a tonal inversion occurs through the developing process of 
a photographic image. In the caricature, the mid-tones and shadow 
areas become darker while any highlighted areas, in contrast, be- 
come brighter. It is as if the tonal spectrum of the image has been 
compressed and the differences between tones are made more ex- 
treme. Sirovich and Kirby, however, conclude that, since the images 


13 Sirovich and Kirby, “Low-dimensional Procedure,” 521. Although Sirovich 
and Kirby were the first to apply PCA to facial images, Turk and Pentland 
were the first to design an automated recognition system utilizing PCA. 

14 Ibid., 519. 

15 Ibid., 523. 
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appear the same to them, they provide evidence that “our own visu- 
al apparatus does a similar subtraction.” In other words, our own 
perceptual processes of recognition most likely involve a process of 
some form of deduction. 

In their recorded observations and conclusions it is possible 
to detect an interesting and altogether separate confrontation 
that occurs in these initial eigenpictures, and that is a confronta- 
tion between human and machine perception. It is exemplified by 
Sirovich and Kirby’s reference to “our own visual apparatus.” For 
the mathematical caricature, the video still and our own process- 
es of perception all represent intersections of perceptual relation- 
ships between multiple apparatuses - photographic, video, com- 
putational and bare human perception - that inform an exercise 
in recognition. With their speculation about the human ability 
to perceive faces and the possibility of expressing this capacity in 
these abstract and mathematically based caricatures, Sirovich and 
Kirby stray into the realm of what John Johnston has described as 
“machinic vision,” that is, “not only an environment of interacting 
machine and human-machine systems but a field of decoded per- 
ceptions that, whether or not produced by or issuing from these 
machines, assume their full intelligibility only in relation to them.” 
Two key movements that Johnston outlines in relation to machinic 
vision are a deterritorialization and a reterritorialization of vision." 
The former occurs when visual perception is freed from the person 
that is doing the seeing, and the latter occurs when that seeing is 
“recoded,” that is, recontextualized and expressed in a new form so 
as to produce new meaning. Sirovich and Kirby’s speculation may 
be understood along these lines: the caricature is an expression of 
vision, in a process of recognition, freed from human cognition. 


16 Ibid. 
17 Johnston, “Machinic Vision,” 27. 


18 Ibid., 28. Here Johnston is drawing on Gilles Deleuze and Félix Guattari’s 
use of the term “machinic.” 
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Figure 5: ATST Laboratories, Eigenfaces of faces from the ORL 
face database. 


Sirovich and Kirby pay close attention to their own visual perception 
of these caricatures. They believe that the conditions of recognition 
can be revealed through them. In this way, the caricatures also re- 
veal the conditions of human recognition. What is also important 
to consider is that, for Sirovich and Kirby, the caricature makes 
visible the human recognition process; in other words, it makes it 
possible to see how we see. In this way, it reveals an “observer,” in 
the sense of the term Jonathan Crary explains in his study of the 
historical construction of vision: “one who sees within a prescribed 
set of possibilities, one who is embedded in a system of conventions 
and limitations.” Crary argues that it is only through the observer 
that vision, in history, is able to “materialise, to become itself visi- 
ble.”° These early experiments with eigenpictures express an early 
assemblage between human cognitive processes of recognition and 
a machinic translation of that recognition. In the caricature, a kind 


19 Crary, Techniques of the Observer, 6. 
20 Ibid.,5. 
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of “machinic observer” is revealed, and we can begin to see how vi- 
sion can be codified within a specific operation and within the con- 
ventions of an operation of recognition. In other words, Sirovich 
and Kirby’s early eigenpictures and caricatures give a visual form to 
the conditions of recognition. 


Three Aspects of Eigenface 


The German prefix eigen- means “inherent, own, individual, peculiar, 
specific, and characteristic.” As this suggests, the facial recognition 
method is supposed to be an algorithmic ability to distinguish what 
is characteristic of an individual’s face in order for the algorithm 
to determine the individual’s identity. The eigenface algorithm is 
designed to do just this. Eigenface is based on the premise that the 
most relevant information about an individual face has to do with 
the ways it is different from another. Eigenface has been successful 
in demonstrating an ability both to detect faces and to encode the 
differences between multiple faces. As Turk and Pentland state, “A 
simple approach to extracting the information [...] is to somehow 
capture the variation in a collection of face images [...] and use this 
information to encode and compare individual face images.” In- 
deed, the primary difficulty in developing a successful AFR system 
is, as Bledsoe had earlier realized, that human faces vary endlessly 
in appearance. The eigenface method takes this difficulty and trans- 
forms it into a recognitive capacity through a tool of differentiation. 
In this way, variation is utilized in encoding an individual face. Yet, 
far from distinguishing particular characteristics, the eigenface 
image depicts a very different process: an erasure of all individual 
facial particularities (figure 5). All that is specific and particular to 
a human face seems to dissolve in a blur. This paradox between the 
method and the image harbors a contradiction in the modalities of 
recognition between algorithm and human. To further elaborate on 


21 “Eigen,” Wiktionary, last modified April 7, 2019, https://en.wiktionary.org/ 
wiki/eigen#German. 
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the mode of recognition by eigenface, I will describe briefly here 
three key technical aspects of the eigenface process that produce 
this image and that constitute its representational mechanism and 
operation of recognition. These three aspects are Principal Compo- 
nent Analysis, the eigenvector and the face space. 

Principal Component Analysis is a statistical procedure that has 
primarily been used as a classification tool and as way of producing 
predictive models based on a statistical method of mean centering. 
In mathematical terms, PCA treats each facial image as a point or 
a vector on a grid with a high-dimensional space allowing for high 
degrees of variation. This high-dimensional coordinate space can 
be understood as Cartesian space gone digital. Each collected fa- 
cial image in the training set is translated into a unit of measure- 
ment, or a weight, within this virtual space. Averages are calculated 
from the different weights of facial images. Each average takes into 
account all the possible variations of each weight. The averaged or 
mean face is described as “the center of gravity for all the faces com- 
bined.”” This averaged face delimits the highest degrees of varia- 
tion, that is, the farthest directions of deviation from the average 
that exist between the collected facial images. Turk and Pentland 
explain that “any collection of face images can be approximately re- 
constructed by storing a small collection of weights for each face.” 
The PCA procedure calculates a mean by averaging the value of each 
pixel across the face images. PCA is able to extract the principal 
components, or the primary differences, between multiple faces 
and encode this variation. Eigenface programmers describe this 
as revealing the internal structure of the data. Sirovich and Kirby 
state: “It seems reasonable to assume that an efficient procedure for 
recognizing and storing pictures concentrates on departures from 
the mean. With this in mind, the deviation or departure from the 
mean is defined.” What they describe is a way of defining the char- 


23 Jeremy Kun, “Eigenfaces, for Facial Recognition,” Math Programming 
(blog), July 27, 2011, https://jeremykun.com/2011/07/27/eigenfaces/. 
24 Turk and Pentland, “Eigenfaces for Recognition,” 73. 
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acteristics of faces through the differences between faces. The mean, 
therefore, becomes a base from which to designate a difference. 

As in many facial recognition algorithms, the application of PCA 
begins with a training set produced from multiple facial images. In 
1986, Sirovich and Kirby were the first to experiment with building 
a training set of faces on which to apply the method of PCA. They 
report taking still analogue shots of video recordings.” Their first 
test group of faces came from their immediate environment, the 
relatively homogenous population of “the undergraduate male pop- 
ulation” that dominated the mathematics department at Brown 
University, a group they describe as made up of “smooth-skinned 
caucasian males.” They recorded the faces for the training set using 
video, on top of which they overlaid a cross-hair aligned vertically 
with the midline of the face and horizontally with the pupils. They 
were able to adjust the depth of field of each video still so that the 
width of each face could be equalized. These images were then dig- 
itized and turned into gray-scale pictures through an image pro- 
cessor. The result of PCA is a mean of all the faces of the training set 
and is depicted in an image in Sirovich and Kirby’s paper (figure 6). 


Figure 6: Sirovich, Kirby, “Average face based on an ensemble 
of 115 faces.” 1987. 


26 Ibid.,522. 
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These initial experiments with training sets and the application 
of PCA are interesting to examine because the average male face 
was constructed from a training set collected from the exclusive- 
ly white, male population of the applied mathematics department 
at Brown University. In choosing faces whose characteristics were 
closer in similarity, it raised the threshold of success for recogni- 
tion. Sirovich and Kirby describe how they purposefully chose faces 
that were similar to each other in order to produce the best out- 
come.” Higher rates of positive recognition are thus dependent on 
a smaller degree of difference between the faces included within 
the modeling of the average face.”* Similarities between the phys- 
ical characteristics of faces is part of the logic of recognition. Even 
at this stage, the building a viable training set, there is already a 
kind of reduction being applied, a normative categorization of faces 
according to characteristics of gender and race. 

There is another technical reduction process that occurs at this 
initial stage. The facial images that make up the training sets are 
converted to gray-scale (if originally in color) and to low pixel res- 
olution. This reduction is indicative of perceptual conditions that 
contrast with those of human processes of recognition. The conver- 
sion to low resolution (usually at 200 x 180 pixels) and to gray-scale, 
that is, to values of light and dark, reduces the amount of visual in- 
formation available to the human eye. Yet, for the algorithm, this 
reduction provides clarity by way of “simplifying” the images — to 
use the vernacular of digital post-production terminology - mean- 
ing that it leaves only the information necessary for the operation 
and gets rid of the rest. Reduction by way of gray-scale and low res- 
olution eliminates the obscurity or extra information that detracts 
from the ingredients the algorithm deems important, that is, what 
can be measured and calculated through pixel values. 

The reduction of the information in the image provides a better 

“palette” for pattern recognition to take place. These initial process- 
es of reduction, both in the format of the images and in the choice 


27 Ibid., 524. 
28 This method made possible improvements to the eigenface algorithm and 
the development of the method of “fisherfaces.” 
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of demographic from which the faces are drawn, are part of con- 
structing the representational mechanism of the algorithm; that is, 
they help shape what is deemed salient and relevant to an opera- 
tion of recognition. In applying statistical processes of reduction, 
PCA performs a kind of transformation of facial variations into a 
“working object.” As Lorraine Daston and Peter Galison explain, a 
“working object” as, “any manageable, communal representative of 
the sector of nature under investigation” such as atlas images and 
type specimens, that at times replace the natural specimens they 
represent.” Organic forms produce endless variation and are un- 
able to “cooperate” in generalizations and comparisons. In contrast 
a “working object” provides a common object inquiry. In describing 
scientific atlases as a “working object,” Daston and Galison explain 
how it served to “teach how to see the essential and overlook the in- 
cidental, which objects are typical and which are anomalous, what 
the range and limits of variability in nature are.” The principle of 
the “working object” was based on allowing for collective scientific 
inquiry to occur through the standardization of natural forms. As 
Daston and Galison show, the creation of working objects was cen- 
tral to the work of scientific inquiry and the classification of natural 
phenomena. PCA performs the task of transforming faces and fa- 
cial variations into a manageable and workable object by defining 
the range of facial variations in an operation of recognition. The 
statistical method of PCA is able to merge multiple natural forms 
of faces into a single conglomerate. As with the working object, 
PCA is able to refine the facial variations and envelop them within a 
readable (at least by a machine) working object of the averaged face, 
transforming the variations into a manageable form. In this way, 
the statistical process of PCA functions like a scientific atlas of the 
algorithm to train it to see the essential aspects of the human face, 
found in the average but also to see the “incidental” or rather the 
deviations from the average as a method of recognition. 


29 Lorraine Daston and Peter Galison, Objectivity (Cambridge: Zone Books, 
2007), 19. 
30 Ibid., 26. 
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Training sets are the data banks and, as such, the source of 
knowledge for an algorithm. They provide knowledge for the algo- 
rithm on what it is allowed to see and recognize. In this way, for an 
algorithm, the training set is the link between knowing and see- 
ing. The training set is what enables the algorithm to know certain 
faces and thus to recognize them. The early examples of images of 
average faces produced by Sirovich and Kirby’s training sets reveal 
a bias toward white men as a primary and normative category of 
recognition. Although the training sets used in their research were 
part of an experiment with eigenface and not examples of the ac- 
tual application of AFR technology, the demographic of the people 
in the images corresponds with that of the people in the training 
sets in the initial experiments with AFR technology conducted by 
Bledsoe. As such, there is a history of the training sets used in the 
development of automated processes of recognition predominant- 
ly involving the figure of the white male. In their analysis of facial 
recognition systems, Lucas D. Introna and David Wood remark that 
such reductions are where bias can be located in the algorithmic 
process.” Although they do not scrutinize specific AFR methods in 
great detail, based on their examination of training databases they 
speculate that, through the reductive process of both image-based 
and feature-based algorithms, minorities are most likely to deviate 
from statistical averages that result from facial recognition systems. 
In analyzing the problematics of reduction, they conclude that mi- 
norities of Asian and African-American descent are the easiest to 
recognize in virtue of this deviation from the mean.” This, they ar- 
gue, contradicts the claims of suppliers and security analysts in the 
biometric industry about the neutrality of AFR systems.” In light of 
Introna and Wood’s findings and the “average faces” presented in 
the papers on the development of eigenface, we may conclude that 
representation in facial recognition systems has been dominated 
by the white male, presenting all other demographics as deviations 
from the norm. 


31 Introna and Wood, “Picturing Algorithmic Surveillance,” 186. 
32 Ibid., 190. 
33 Ibid., 191. 
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The application of PCA to training set images creates “eigenvec- 
tors,” that is, mathematical objects that display the degree of vari- 
ability or deviation between facial characteristics and an average. 
Each eigenvector represents the greatest degree by which the facial 
images may vary - i.e., the highest eigenvalues. Multiple eigenvec- 
tors result from applying PCA to training sets, creating a mecha- 
nism to classify unknown faces based on their deviation from these 
eigenvectors. The eigenvector is a virtual model of “known” faces 
and serves as a reference point for the classification of unknown 
faces. Like a map, an eigenvector stands as a kind of idealized mod- 
el; it acts as a primary referent on the basis of which the algorithm 
is able to measure the distance or variation between it and an un- 
known face. As a virtual model, the eigenvector is a form of repre- 
sentation that transforms the pictorial, individual representations 
of known faces into a geometrical space defined by facial measure- 
ments. In this way the eigenvector comes to represent faces based 
solely on their relationships to other faces. The eigenvector is a rep- 
resentation of the differences and similarities between faces and in 
this way functions as a unit of facial measurement. 

It is only when an eigenvector is displayed to meat eyes, that is, 
to human vision, that it is referred to as an eigenface. The greater 
the variation of an eigenvector, the more blurred the eigenface ap- 
pears. Programmers have referred to eigenvectors as “ghost faces” 
because of their phantom-like appearance. These programmers are 
describing the form of these eigenvectors, which is characterized 
by the multiplicity that is inherent in the statistical process. The 
programmers’ reference to ghost faces evokes a notion of imagery 
put forward by W. J. T. Mitchell, who describes a type of imagery 
that is perceptual and occupying, “a kind of border region where 
physiologists, neurologists, psychologists, art historians, and stu- 
dents of optics find themselves collaborating with philosophers 


34 Mige Carikci and Figen Özen, “A Face Recognition System Based on Eigen- 
faces Method,” Procedia Technology 1 (2012): 122, https://doi.org/10.1016/j. 
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and literary critics.”* He further describes this imagery as playing 
the role of fantasmata and as existing as “revived versions of those 
impressions called up by the imagination in the absence of the ob- 
jects that originally stimulated them.” The eigenface image can be 
understood as this type of fantasmatic image. Indeed, one of the 
primary aesthetic features of the eigenface image is the absence of 
the individual face, which disappears in the midst of its conglom- 
erate form. Instead, the facial appearances in the eigenface image 
function symbolically to construct the virtual facial model, which 
acts as the central referent in the recognition process. 

The collection of eigenvectors create a subspace referred to as 
the “face space.” Eigenface developers have described the face space 
as a virtual subspace that is defined and framed by the measured 
distances between a collection of eigenvectors. The concept of a face 
space derived from the field of psychology in interpreting how fac- 
es are processed by human recognition. The face space is originally 
defined as a, “multidimensional psychological space, in which fac- 
es can be represented according to their perceived properties” and 
with the, “assumption...that faces (or concepts) could be represent- 
ed as a collection of interchangeable parts.” The face space in the 
eigenface algorithm is defined by the range of variability of these 
“interchangeable parts.” Conceptually, the face space spans all pos- 
sible variations of faces. Turk explains that any kind of: 


image deviations (whether due to image noise or other factors, 
such as illumination, pose, expression, occlusions, etc.) push an 
image away from the space, and the distance from face space can be 
used to determine how likely an image is to be a face in the first 
place, thus providing a built-in mechanism for face detection.2” 


35 WJ. T. Mitchell, Iconology: Image, Text, Ideology. Chicago: University of 
Chicago Press, 1986, 10. 

36 Tim Valentine, Michael B. Lewis, and Peter J. Hills, “Face-Space: A Uni- 
fying Concept in Face Recognition Research,” Quarterly Journal of Experi- 
mental Psychology 69, no. 10 (2016): 1996-2019, https://doi.org/10.1080/1747 
0218.2014.990392. 
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Turk’s description thus identifies the face space as the source of 
facial detection in the algorithm. The face space operates as a vir- 
tual data bank, storing an algorithm’s knowledge, enveloping all 
the variations of possible faces that the eigenface algorithm can 
conceivably recognize. The collection of eigenvectors with highest 
eigenvalues, that is, highest measured variations, is used as a refer- 
ential source for algorithmic knowledge. In this way, the face space 
is like a virtual filing cabinet - the primary bureaucratic mecha- 
nism of identification in criminology. But instead of a filing cab- 
inet storing individual records, the face space collects statistical 
averages or eigenvectors to serve in the process of recognition and 
verification. 

The actual recognition process in eigenface involves projecting 
the captured image of the individual who needs to be identified on 
to the face space. The image is compared with the face space by cal- 
culating the Euclidian distances between the eigenvectors and the 
captured image. If there is a small distance between the capture 
and the eigenface, there may be a match. The distances between 
them are then expressed in numerical values and a data set is creat- 
ed. This data set then represents a person’s identity and is entered 
into a database. An individual is classified within a biometric data- 
base not through their image but rather through numerical code. If 
there are large distances between the capture and the eigenvectors 
within the face space, then there is no match. If there is no match, 
the captured face can be incorporated into the algorithm, adding a 
new variation within the eigenvector. Turk and Pentland expand- 
ed on Sirovich and Kirby’s use of the PCA method by incorporat- 
ing this machine-learning technique within the eigenface method. 
They state: “The concept of face space allows the ability to learn and 
subsequently recognize new faces in an unsupervised manner.”* 
Turk and Pentland describe the process by which the detection of 
unrecognized faces builds new patterns in the algorithm: 


When an image is insufficiently close to face space but is not classi- 
fied as one of the familiar faces, it is initially labeled as “unknown.” 


38 Turk and Pentland, “Eigenfaces for Recognition,” 79. 
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The computer stores the pattern vector and the corresponding un- 
known image. If a collection of “unknown” pattern vectors cluster 
in the pattern spaces the presence of a new but unidentified face 
is postulated.?? 


The statistical average or eigenvector defines the recognition proce- 
dure and is the form of measurement in relation to which a face is or 
is not recognized. In machine learning, the algorithm is designed 
to continue developing over time as facial variations are continu- 
ously added and learned. 


Figure 7: Vincent Scheib, “Eigenfaces of UNC”. 


Face spaces provide visual sources of information about the eigen- 
face process. As collections of eigenvectors, face spaces depict a 
wide spectrum of measured distances of similarity and difference. I 
present three face spaces here in order to illustrate the kinds of vari- 
ation they exhibit, as well as the issues and problems that arise in 
the construction of these spaces. By presenting eigenvectors side by 
side, face spaces can illustrate specific aspects of variation in these 
eigenvectors. The face space in figure 7 is typical of the face spac- 
es created by programmers working with eigenface. One thing that 
we can see clearly in this face space is the variation between the dif- 


39 Ibid. 
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ferent eigenvectors.*° The eigenvector at the top left has the highest 
eigenvalues, representing the average of all the 201 faces within the 
training group. The eigenvector immediately to the right captures 
the overall brightness of the face in the picture. The two following 
eigenvectors to the right capture the direction from which the face is 
illuminated. The eigenvectors on the bottom row capture variations 
in face shape. As the programmer states, the rest of the eigenvectors, 
of which there were 201, capture more subtle details. 


Figure 8: Alexandra Feldman, “Face Recognition: Final Project CS 129, 
Spring 2011,” Computer Science at Brown University 


Eigenface 4 Eigenface 5 Eigenface 6 


The greater the variation of an eigenvector, the more blurred it ap- 
pears. For example, figure 8 is a face space created by a computer 
science student at Brown University.” This face space stood out to 


40 Vincent Scheib, “Eigenfaces of UNC,” accessed April 25, 2015, http://www. 
scheib.net/school/uncfaces/index.html. 

41 Alexandra Feldman, “Face Recognition: Final Project CS 129, Spring 2011,” 
Computer Science at Brown University, accessed April 25, 2015, http:// 
cs.brown.edu/courses/csci1290/2011/results/final/amfi/. 
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me from other face spaces because each individual eigenvector is 
constructed through extreme contrasts of light and shadow, cre- 
ating a grouping that is aesthetically characterized by a painterly 
effect, where each conglomerate facial feature resembles a brush 
stroke. Rather than the usual blur, the facial features are defined 
through blunt markings of light and shadow. The eyes of the figure 
in the upper-middle eigenvector are blacked out, like two large ink 
spots, while the eigenvector in the lower left sports a glowing beard. 
From a programming perspective the student who constructed 
this face space was considered to have made a gross error (and was 
required to fix the contrast levels in her algorithm), its tonal ex- 
tremities reveal an aspect of the process by which face spaces are 
built. Increasing the contrast levels so dramatically causes the fea- 
ture similarities within each eigenvector to become more distinct 
- so much so that these distinctions begin to merge into each other 
and construct other (facial) forms out of the composited similar- 
ities. The eigenvector in the upper-left corner is rendered almost 
completely black. This eigenvector has the highest eigenvalues in 
the face space. The higher value, which equates to a larger amount 
of information for machine perception, amounts, paradoxically, to 
the least visually coherent image for human vision. 


Figure 9: Wonju Lee, Minkyu Cheon, Chang-ho Hyun, and Mignon Park, 
“Best Basis Selection Method Using Learning Weights for Face 
Recognition,” Sensors (Basel) 13, no. 10 (October 2013) 
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Figure 9 is a face space included in an article written by four en- 
gineers who were experimenting with an alternative method for 
selecting eigenvectors for face spaces. In this method, the eigen- 
vectors were selected not on the basis of their eigenvalues but rather 
through groupings of similar, closely related faces.” The face space 
thus created highlights the extreme differential values produced by 
misaligned faces and is expressed through varying tones of illumi- 
nation. The silhouette of differently positioned faces creates a halo 
effect. These “halos” consist of light and shadow that correspond to 
high eigenvalues. As in the previous face spaces, the eigenvalues 
descend in order, starting in the top-left corner. The students who 
created this face space argue that the eigenvectors in group (a), the 
three images on the top row, are “unimportant eigenfaces” because 
they do not “have discriminant information™ that would allow the 
technology to perform an act of recognition. They argue that the 
faces in the training set should be cropped even closer so as not to 
include the illuminated edges within the eigenvalues of an eigen- 
vector. Their rationale here is that the variation in illumination was 
found to have weakened the overall recognition process, and their 
aim was to find ways to more tightly define facial variation in order 
to allow for more precise techniques of recognition. 


The Eigenface Image 


I have outlined these three technical aspects, the PCA, the eigenvec- 
tor and the face space, because they structure the ways in which the 
algorithm performs recognition. They also form the empirical ba- 
sis for my own analysis of how the algorithm perceives. These three 
aspects provide routes into an understanding of the conditions of 
recognition in an AFR system. They explain how a face comes to be 


43 Wonju Lee, Minkyu Cheon, Chang-ho Hyun, and Mignon Park, “Best 
Basis Selection Method Using Learning Weights for Face Recognition,” 
Sensors (Basel) 13, no. 10 (October 2013): 12830-51, https://doi.org/10.3390/ 
$131012830. 
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known and how information is produced and archived by the al- 
gorithm through successful operations of recognition. In order to 
understand these processes within the cultural and socio-politi- 
cal contexts in which they take place, I plan not to examine their 
more technical elements but instead to place them into a dialogue 
with ideas from the field of visual culture theory and to recognize 
them as enculturing an algorithmic way of seeing and medium of 
thought. What I mean by “medium of thought” here is that eigen- 
face is not a static technical process; it is rather designed to learn 
new faces continuously and to incorporate them within its face 
space. The representational mechanism of PCA and the production 
of eigenvectors are the eigenface algorithm’s means of knowledge 
accrual. The eventual operation of recognition produces knowl- 
edge of an individual’s identity. In this way, the algorithmic way 
of seeing is intertwined with modes of contemporary knowledge 
production. These technical aspects thus reveal the ways in which 
the algorithm sees and knows. John Berger opens his seminal book 
Ways of Seeing by outlining the intimate relationship between seeing 
and knowing: “Seeing comes before words [...] The relation between 
what we see and what we know is never settled.” Berger describes a 
productive and “always-present gap” between knowing and seeing 
that is fundamental to the experience of visual perception.** In con- 
trast, seeing by way of recognition through an automated algorith- 
mic process reverses the order Berger describes. Knowledge comes 
before seeing. In this section, I examine this reversal by asking how 
it affects the subject being seen and by tracing the ways of seeing 
present in the aspects of the eigenface algorithm discussed above. 
The production of the eigenface image marks a kind of pictori- 
al turn, albeit a brief one, in the development of automated facial 
recognition technology. In this turn, images come to supplement 
equations in the operations of the algorithmic process. The eigen- 
face image is a visualization of a statistical process. It depicts a sta- 
tistical way of seeing in which the eigenvector, or the aggregate face, 
becomes the source of knowledge for the algorithm. Statistics, as a 


45 Berger, Ways of Seeing, 7. 
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way of seeing, is not new. In fact, statistics has been described as a 
tool of visualization, as a way of seeing on a scale beyond the capac- 
ity of human senses. In his critical genealogy of the term “informa- 
tion,” media theorist John Durham Peters describes how statistics 
merged with an understanding of information in contemporary 
society. Peters notes that the etymological origins of the word lie 
in the German statistik, meaning the comparative (and competitive) 
study of states. Statistics arose as a tool of politics and state gov- 
ernance. Peters states, “The scale of the modern state presents its 
managers and citizens with a problem: it is out of sight and out of 
grasp. It must be made visible.” He goes on: “statistics arose as the 
study of something too large to be perceptible — states and their cli- 
mates, their rates of birth, marriage, death, crime [...] [Statistics is] 
a set of techniques for making those processes visible and interpre- 
table.™* As such, statistics is a visual tool that is inextricably bound 
up with its original context of use: state governance and control. 

Peters highlights an analogy between the acquisition of knowl- 
edge and the body, describing statistics as the “empiricism of the 
state,” whereby “the state becomes the knower, bureaucracy its 
senses, and statistics its information.” When information comes 
to be understood in the form of statistics, the process of knowledge 
accumulation relocates from the site of the body to the site of the 
governing institution. Peters characterizes information produced 
through statistics simply as “knowledge with the human body tak- 
en out of it.” Considering the use of the statistical method of PCA 
as the representational mechanism of eigenface, this suggests an 
ontological paradox. The eigenface method, and other AFR meth- 
ods, are wholly reliant on statistical pattern recognition process- 
es to produce information, but this information is constituted by 
and issues from the body itself. This presents us with a situation in 
which the physical phenomenon deemed invisible and ungraspable 
is none other than the body itself. 


47 Peters, “Information,” 14. 
48 Ibid. 

49 Ibid. 

50 Ibid.,15. 


Chapter 2: Eigenface 


The foregoing discussion about statistics as a tool of vision rais- 
es the issue of the treatment of the body as information. Btihaj Aja- 
na, a scholar working within the field of digital cultures, sets out an 
important approach to the understanding of the body as informa- 
tion with regard to biometric practices. Drawing on Eugene Thack- 
er’s concept of “biomedia,” she argues for an understanding of the 
use of biometrics as “less as a tool and more as a process, less as an 
instrument and more as an act.” Along these lines, Ajana argues 
that the result of these biometric processes and acts is that the body is 
rendered as “both the ‘medium’ (the means by which ‘measurement’ 
is performed) and the ‘mediated’ (the ‘object’ of measurement).”” 
Through the differential calculations of the eigenvector, the recog- 
nition process in the eigenface method realizes the convergence be- 
tween these two roles that the body plays. Irma van der Ploeg claims 
that the treatment of the body as information in biometric practic- 
es introduces a new body ontology. She describes a process of the 
“informatization of the body,” in which bodies are represented in 
digital code, which “construe[s] the body in terms of flows of infor- 
mation and communication patterns.” She describes the body as a 
historical construction (much as Peters describes information) that 
is “implicated in a process of co-evolution with technology — infor- 
mation technologies, but also surgical, chemical and genetic and vi- 
sualization techniques, and combinations of these.” She describes 
how biometrics produces new forms of knowledge that transform 
our understanding not only of technology but of the body itself. 

The eigenface image, as a visual artifact of the algorithmic 
process, allows us to investigate the visuality of the informatized 
body of AFR systems. The image presents us with the way faces are 
depicted, read and treated as information. The visualization of its 
patterns and form is a result of a statistical representational mech- 
anism, PCA. Algorithms are often invisible, operating in a “black 
box” and leaving no trace of their processes of computation behind. 


51 Ajana, Governing through Biometrics, 23 (italics in original). 
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The algorithm does not need to produce pictures to understand its 
own process of recognition. The picture is for human eyes. The ei- 
genface image serves as a training image, not for the algorithm but 
rather for us; it allows human eyes to see like algorithmic eyes. As a 
window into this process, the image answers one of the sociological 
criticisms of AFR systems, namely, that they lack reciprocity — its 
systems identify people without identifying with people. The eigen- 
face image provides a form of visual reciprocity, depicting what and 
how the algorithm “sees.” But observation of the eigenface image 
provides the very opposite of clarity concerning a person’s identity. 
The eigenface image depicts a moment of stasis between the mul- 
tiple inputs of data from the training set and the singular output 
of recognition. The images compiled in the eigenface training sets 
offer a multitude of possibilities and a wealth of variation, while the 
operation of recognition reduces the output to one possible match. 
Positioned at this in-between phase of the algorithm, the eigenface 
image not only presents a statistical process but also preserves a 
moment at which multiple possibilities remain open. 

Turk has explained that he and Pentland designed the produc- 
tion of the eigenface image as part of the algorithm because they 
“wanted to keep clear of the ‘black box’ approach of [...] neural net- 
works [...] in order to have a better ability to understand and debug 
the method.” The eigenface image is part of the recognition oper- 
ation; it is really information and not an image in the traditional 
sense. It is an example of what the artist Harun Farocki has termed 


na 


an “operational image,” “the aesthetics of which were not intended 
[...] instead of representing the objects in the world, these images 
are doing things in the world, they are part of a process [...] they 
are information and not really images.”* Operational images are 
produced by a machine and are self-reflexive in the sense that they 
depict both the conditions of observation and what is observed by 
the machine. In this way, the eigenface image is, in a sense, pure 
information. Yet, for the human observer, it gives a sense of a mo- 
dality of machinic recognition, complete with an inherent aesthetic, 


as is suggested by its description as a “ghost face.” In developing the 


55 Farocki, War ata Distance. 


Chapter 2: Eigenface 


concept of the operational image further, specifically with regard to 
surgical imaging, Aud Sissel Hoel and Frank Lindseth say that they 
are “generative,” that they “differentially intervene,” distilling char- 
acteristic patterns that would not be seen in other ways.” The eigen- 
face image can also be read in this way, as depicting a process of the 
statistical pattern recognition of multiple faces through a process of 
differentiation that results in an operation of recognition. 

Sirovich and Kirby claim at one point in their paper that the 
“eigenpicture” is a way of “making matters more concrete.” This 
claim about the function of the eigenface image relates to its role 
as a technical image, as Vilém Flusser terms images that structure 
information in contemporary society and replace other forms of 
communication. Flusser differentiates the technical image from 
what he calls “traditional images.” Technical images, he says, 
operate at the “intervals” of understanding. They “translate par- 
ticles” and “bits of information” that could otherwise not be seen 
into something that is “graspable, conceivable, tangible.” Flusser 
says that one of the functions of the technical image is specifically 
to make graspable information that has become abstract through 
processes of mathematical calculation. Importantly, Flusser de- 
scribes the technical image as having an ability “to turn from 
extreme abstraction back into the imaginable.” The concepts of 
the operational image and the technical image offer ways of un- 
derstanding the role of the eigenface image within the context of 
its production, as a part of the operation of recognition. But they 
also offer a way out: that is, they draw our attention to how the ei- 
genface functions outside of the programming context and within 
a wider ecology of images that break from traditional notions of 
representation. In particular, two features of images are useful in 


56 Aud Sissel Hoel and Frank Lindseth, “Differential Interventions: Images 
as Operative Tools,” Photomediations: A Reader, ed. Kamila Kuc and Joanna 
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approaching the eigenface method within the context of notions 
of recognition and identity: their “generative” character and their 
ability to render things “imaginable.” 


Chapter 3: Francis Galton 
and the Composite Portrait 


The computer existed as a prac- 
tice before it existed as a ma- 
chine. 

Lewis Mumford, “The Myth of 
the Machine” 


Pictorial Statistics 


One of the things that becomes clear once we shift our approach to 
the eigenface image from treating it as a computational object of 
study to an object of study in visual culture is that, as an image, it 
has a history. Just as Peters traces the merging of statistics with a 
contemporary understanding of information, here I seek to trace 
the merging of statistics with vision that has occurred through 
practices of facial recognition. I do so in order to reveal the specif- 
ic cultural and socio-political contexts in which this merger came 
about and developed. This approach runs counter to the assump- 
tions of neutrality and objectivity made on behalf of the technology 
and instead relates the technical processes of facial recognition to 
discourses concerning representation. In the 1880s, British anthro- 
pologist, statistician and founder of the theory of eugenics Fran- 
cis Galton began experimenting with what he called the composite 
portrait.! Galton’s practice of composite portraiture can be under- 


1 Galton gave numerous talks on his practice of composite portraiture and 
published articles in Nature. He also described the processes and conclusi- 
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stood as an antecedent of the representational mechanism used in 
the eigenface algorithm.” The process by which Galton construct- 
ed his composites was painstaking - a kind of repetition of acts 
through which the variance of the human hand could fall away. Gal- 
ton would stack photographic portraits on top of one another and 
hang them on the wall in front of the camera (Figure 10). 


Figure 10: Francis Galton, “Composites assembled for photography” 
in Popular Science Monthly Volume 13, (August 1878). 


The lighting and scale in each portrait had to be identical, with each 
face in the same, forward-facing position. Galton made a physical 
crosshair out of thread, placing it in front of the stack of portraits 
hanging on the wall. The crosshair ran horizontally through the 
center of the eyes and vertically through the midline of the face in 
each image. (Figure 11) 


ons of his practice in length in his work Inquiries into Human Faculty and Its 
Development, 2nd ed. (London: Dent, 1907; galton.org 2018), http://galton. 
org/books/human-faculty/SecondEdition/text/web/human-faculty4.htm. 


N 


Interestingly, the inventor of PCA was mathematician and biostatistician 
Karl Pearson, who was a protégé of Francis Galton and was a central ear- 
ly figure in the study of biometry and social evolution. Yet his invention of 
PCA, in 1901, was unrelated to his studies of social evolution or to Galton’s 
practice of composite portraiture. See Karl Pearson, “LIII. On Lines and 
Planes of Closest Fit to Systems of Points in Space,” The London, Edinburgh, 
and Dublin Philosophical Magazine and Journal of Science 2, no. 11 (1901): 559- 
72. 
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Figure 11: Francis Galton, “Step one in assembling a composite photo- 
graph” in Popular Science Monthly Volume 13, (August 1878) 


He then took a picture of each facial image using a single photo- 
graphic plate, closing the aperture of the camera in between each 
shot, when he would turn to the next image in the stack by hand. 
The result was that he exposed a single photographic plate to all the 
photographic portraits hanging on the wall. The exposure time for 
each photograph was calculated as a fraction of the total exposure 
time for the photographic plate. For example, if there were ten fa- 
cial images and the photographic plate had to be exposed for a total 
of a twenty seconds, each facial image would be exposed for two 
seconds. 

Galton originally applied the composite technique to pho- 
tographs of landscapes to track changes in topography. He ap- 
proached images of the human face as unknown landscapes. The 
late nineteenth century witnessed a seemingly inexorable growth 
in the urban population. Just as the overwhelming scale of the 
modern state, according to Peters, raised issues of visibility and 
knowability, so this population growth created a supposed need to 
recognize certain segments of this population, such as criminals 
and other “unknowns,” in order for them to “be made visible,” and 
this supposed necessity informed Galton’s practice. His composite 
portraits employed statistics in order to visualize that which would 


87 


88 


Portraits of Automated Facial Recognition 


otherwise go unseen; they were a statistical table in pictorial form. 
Galton stated of his composites that “they are rather the equivalents 
of those large statistical tables whose totals, divided by the number 
of cases, and entered in the bottom line, are the averages. They are 
real generalisations, because they include the whole of the mate- 
rial under consideration.” Galton created composites of many so- 
ciologically defined groups, including portraits of those who had 
committed specific crimes, groups of people with particular med- 
ical ailments, people from various ethnicities and members of the 
Jewish population, as well as producing pictures in accordance with 
idealized categories of beauty and intelligence. 

For Galton, the composite’s ability to construct representational 
faces promised a multitude of operational possibilities. Construct- 
ing pictures of characteristic physiognomic traits could function 
as a form of identification and social control. Ultimately, Galton’s 
operation of composite portraits was a failure.* Yet the visual prac- 
tice of composite portraiture and what it reveals continues to be a 
source of study and allure today for artists, scholars and scientists. 
There have been numerous studies and analyses of Galton’s com- 
posite portraits, which testifies to their ability to continue to evoke 
curiosity and prompt experimentation in many different fields of 
study. Galton’s use of photography made it possible to freeze a vi- 
sual process of abstraction. In their book Objectivity, Lorraine Das- 
ton and Peter Galison describe the privileged position of the cam- 
era in the history of scientific imagery: “machine-regulated image 
making was a powerful and polyvalent symbol, fundamental to the 
new scientific goal of objectivity.” As they explain, the use of the 
camera supported a turn, in the 1830s, toward a scientific “devotion 
towards depicting what was seen on the surface, not what was de- 
duced or interpreted.” Additionally, as a visual apparatus, it held a 


3 Galton, Inquiries, 233. 

4 See Elizabeth Stephens’s article on the productive nature of this failure. 
“Francis Galton’s Composite Portraits: The Productive Failure of a Scientif- 
ic Experiment,” unpublished manuscript, June 2013, https://www.research 
gate.net/publication/323275029_Francis_Galton’s_Composite_Portraits_ 
The_Productive_Failure_of_a_Scientific_Experiment. 
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“promise of automaticity” in depiction and perception - much like 
the perceptual algorithms of the present day - which could produce 
judgment-free reproductions of physical phenomena. Its machinic 
abilities were thus equated with a level of authenticity. 

Galton describes the composite as depicting a thought process. 
The advantage of the camera, for Galton, lay in its ability to visu- 
ally represent the abstracting process of statistics in the form of 
facial images. With the practice of repetitive exposure, that is, in 
multiplying the reproduction process by photographing the pho- 
tograph, Galton’s composites transformed the use of the imaging 
apparatus into a form of statistical measurement. Galton’s use of 
the camera to apply the abstractive process of statistics resulted in 
a depiction that, as Daston and Galison put it, “passed from indi- 
vidual to group.” The use of the camera made it possible to repeat- 
edly reference the same, singular portrait in multiple composites. 
This aspect also allowed for the application of statistical methods to 
photographic imagery. It also made it possible to construct an im- 
age based on an underlying logic of inductive inference, that is, the 
act of observing many specific cases in order to discern a pattern 
and deduce a general idea. For Galton, mechanizing the abstractive 
procedure through the use of the camera provided a perfect way 
to merge both apparatuses - the statistical and the photographic — 
and thus to make abstract phenomena visible, factual and concrete. 
The pictorial form of the statistical found in the composite photo- 
graph made the generalizations these images contained visible not 
only to Galton but, as he himself noted, to anyone looking at the 
composite — a collective vision. This feature made the composite 
image useful for those guided by the ideals of science.’ 


6 Ibid., 170-71. 
7 Galton, Inquiries, 233. 
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Zones of Normality 


There are two significant influences that have been mentioned in 
relation to Galton’s approach to the production of composite por- 
traits.* These influences are the theory of abstraction put forward 
by the philosopher John Locke and a theory of social statistics of 
which the Belgian sociologist Adolphe Quetelet was the primary 
proponent. These two influences are pertinent to an analysis of the 
logic of recognition and its connection to the eigenface method. 
Firstly, Galton’s approach to composites was based on a desire to 
create representational faces. His interest in representational fac- 
es grew out of a philosophical question about the construction of 
general ideas that originated in Locke’s theory of abstraction. Locke 
examined this question in relation to language and the formation 
of words, that is, signs that represent general ideas. Locke claimed 
that general ideas are formed by the mind separating them out 
from the particulars of context, time and place, which makes it pos- 
sible for these ideas to represent more than one individual case. In 
brief, Locke believed that generalizations can be formed by simply 
leaving out what is particular to each case. Galton took up this line 
of thinking in relation to his studies of physiognomy, the “science” 
of judging a person’s character, behavior and personality based on 
their physical features. Physiognomy often focused on the face as 
the part of the body that could supposedly reveal behavioral char- 
acteristics that lay underneath. Galton applied the theory of ab- 
straction to faces, attempting to create a general face as a sign that 
could stand for a group. Galton developed his own theory, which 
he coined “eugenics” — he was the first to use the term — based ona 
belief in a kind of rascist genetic determinism. 

Galton’s practice of the composite image was situated within 
a specific socio-historical context. Allan Sekula describes Galton’s 
work with the composite portrait as grounded in the codified theory 
of “social statistics,” which focuses on social behaviors that can only 


8 Thanks to James Conant for discussions about his research into the philoso- 
phical underpinnings of Galton’s composite portraiture practice. 
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be observed and identified through statistical means.’ A central 
conceptual category in social statistics is that of “the average man,” 
a notion that comes from Quetelet. Quetelet depicted this concept 
in the form of a statistical graph, a bell curve in which the average 
lies at the apex (figure 12). Sekula explains that Quetelet came to 
regard this pattern as “the mathematical expression of fundamen- 
tal social law,” essentially treating regular occurrence as evidence 
of a truth." Most importantly, Quetelet defined the central portion 
of the curve as a “zone of normality” such that deviations too far 
from the apex would fall into categories of “biosocial pathology” 
and “monstrosity.” 


Figure 12: Adolphe Quetelet, “Binomial distribution, 999 trials, 
histogram,” in Lettres sur la theorie des probabilites appliquée aux 
sciences (Bruxelles, M. Hayez, 1846): 103 


l M 
Hie 


Particular social groups of the population were thereby defined as 
categories through their deviation from the zone of normality. Here 
we find an origin, based on statistical logic, for the normative cat- 
egories that define a process of recognition. Sekula describes the 
formation of the concept of “the average man” as based in part on 


9 Robert A. Nisbet, “Social Science,” Encyclopedia Britannica, September 27, 
2018, https://www.britannica.com/topic/social-science/The-20th-century. 

10 Allan Sekula, “The Body and the Archive,” October 39 (Winter 1986): 19. [3- 
64] 

11 Ibid. 22. 

12 Ibid. Sekula is here quoting Adolphe Quetelet, Lettres sur la théorie des pro- 
babilities appliquée aux sciences marales et politiques (Bruxelles: M. Hayez, 
1846). 
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aesthetic values, borrowing from “art historical evidence of noble 
Grecian foreheads” and a “racist geometrical fiction.”” 

Deviations from a norm created another set of sociological cat- 
egories. Particular medical ailments and criminal behaviors were 
societal deviations from a norm, and, as part of his eugenics project, 
Galton wanted to relate typical cases of these to sociological theo- 
ries of heredity. The eugenics project was taken to its eventual ex- 
treme in its ideological and genocidal adoption by the Nazi regime 
in Germany. As part of an attempt to construct a cultural aesthetic 
for the regime, the Nazis’ infamous art exhibition “Degenerative 
Art” was held in Munich in 1937, exhibiting pieces of modern art, in- 
cluding works of expressionism and cubism - what Hitler referred 
to as “the isms.” Along with the exhibition, a catalogue was printed. 
The basis for this catalogue was a book by Paul Schultze-Naumburg 
titled, “Kunst und Rasse” (Art and Race).“ Schultze-Naumburg’s 
book exemplified how the relationship between aestheticized so- 
ciological norms and their deviations can be depicted through art. 
It included photographic portraits of individuals with degenerative 
medical ailments that left them with deformed faces. These were 
placed side by side with modernist, painted portraits. Works by 
Karl Schmidt-Rottluff, Otto Dix and Amedeo Modigliani served as 
examples that looked akin to photographic portraits of disfigured 
faces. This stereoscopic comparison between photographs and 
paintings was utilized to associate modernist art with deviant so- 
cietal behavior and ill-health in the human population, essentially 
accusing modernist art itself of being a sociological deviation. The 
Nazis interpreted the paintings as direct illustrations of a person’s 
physical, surface deformities, rather than expressions of the sub- 
jective emotional states of the artists. The coupling of these works 
expresses an interesting juxtaposition between two types of image 
production: the expressiveness of the artist and the mechanical pre- 
cision of photography. Both provided the Nazis with expressions of 
supposed deviations from a norm or ideal. Quetelet’s statistical- 


13 Sekula, “Body and the Archive,” 22. 
14 Schultze-Naumburg, Paul. Kunst und Rasse, Munich, New York: J.F. Lehmanns 
Verlag, 1928. 
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ly based concept of a sociological norm provided the justification 
for the notion of deviation. In opposition to this “degeneration” in 
both art and the human race, the Nazis preferred the “great” Ger- 
man artworks, filled with pieces capturing the physical ideals of 
the “Aryan” race. This provides an extreme example in which the 
categories set out in social statistics theory constructed a notion of 
modern identity. 

Both the theory of abstraction and social statistics influenced 
Galton’s descriptions and perceptions of the “type” that emerged 
from the layers of his composite portrait. The significance of the 
composite portrait, for Galton, lay in its ability to depict what is 
statistically salient, the “type.” Galton described his composites as 
a “portrait of a type and not of an individual [...] an imaginary fig- 
ure possessing the average features of any given group.” As Seku- 
la points out, “type” and “typicality” were words used by Quetelet 
and closely tied to his application of statistics and the promise of 
sociological truth. The word “typical” expresses the same idea as 
the word “generic,” with the latter presupposing the existence of a 
genus. Galton described a type as “a collection of individuals who 
have much in common and among whom medium characteristics 
are very much more frequent than extreme ones.” Galton gathered 
faces for the composites based on what he believed to be common 
physical traits among a group of persons, and these groups, in turn, 
were based on his own sociologically constructed categories of eth- 
nicity, race, medical or criminal history and social status. The re- 
sulting composites were a way for Galton to visually represent the 
general norm among these groupings such that the particularities 
of each individual would fall away. Galton described his own per- 
ception of the composites as follows: “All that is common remains, 
all that is individual tends to disappear.”” In this way, the type 
functioned for Galton as the word functioned in the formation of 
language in Locke’s theory of abstraction: the typical face stands 
in as a general sign for multiple faces of the same group (Figure 13). 


15 Galton, Inquiries, 222. 
16 Ibid., 230. 
17 Ibid. 
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Figure 13: Francis Galton, lis ala Portraiture, 1883 
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Thus, for Galton, the particular instances faded into the blur of the 
image: 


Those of its outlines are sharpest and darkest that are common to 
the largest number of the components: the purely individual pecu- 
liarities leave little or no visible trace. The latter being necessarily 
disposed equally on both sides of the average, the outline of the 
composite is the average of all components." 


In this description we can hear Galton adopting much the same ap- 
proach to interpreting his image as Quetelet did to his graph of the 
“average man” and the corresponding “zone of normality.” Like Gal- 


18 Ibid., 223. 
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ton’s composite, the pinnacle of the bell curve in Quetelet’s graph is 
where clarity and statistical salience lies, with its height depicting 
the mean. The center of the composite portrait is, for Galton, the 
significant part of the image. Just as Quetelet had described how, 
“on both sides of the average,” the individual instances decrease and 
fade, so, for Galton, on all sides of the center of the composite, the 
particularities fade. At the center of both the graph and the pho- 
tographic composite you have what is considered by Galton to be a 
clear impression of commonality, whereas the extremes or devia- 
tions from that commonality fade out from the center. The statis- 
tically salient “type” is perceived as a visible space in which there 
is “agreement” and where all the “irregularities disappear, and the 
common prevails.” The “type” seen at this central mean is able to 
transcend the blurriness of particulars - all that is rendered un- 
graspable and unseen by excessive variation — and rise above as a 
representative face, a face that is able to clearly stand for the char- 
acteristics of a given group and, as Galton so succinctly observes, 
“erase the ghostly blur of difference.” 

Galton describes the type not as something that is perceived on 
the surface of the image but rather as something that becomes visi- 
ble, from beneath the layers of faces, through a process of perceptu- 
al emergence. This perceptual emergence was facilitated by certain 
aesthetic factors with which Galton experimented. For example, he 
observed that composites are best seen in miniature. In a smaller 
format, the type may more easily be perceived by an observer. One 
of the more ambitious aspects of Galton’s composite portraiture 
was his use of the photographic medium - a medium grounded in 
indexical meaning — for the purpose of producing a representation 
of a face that does not exist in physical reality. Galton’s production 
of the composite utilized the photographic medium to produce a 
model. The process of repetitive exposure in Galton’s composite 
portraiture transformed the photographic apparatus into a kind of 
statistical measuring device. The emergence of a perceivable “type” 
materialized a distinct figure of the imagination on the basis of 
which it is possible to recognize individual members of the group. 


19 Ibid., 224. 


95 


96 


Portraits of Automated Facial Recognition 


Thus, this photographic construction of a type is an early anteced- 
ent of the virtual models used in AFR technology. 


Figure 14: Francis Galton, “Indian Portraits of Alexander the Great with 
Composite in centre” in Karl Pearson, The Life, Letters and Labours of 
Francis Galton Vol 2, 1924: 296 


A Sociological Ideal 


Galton’s composite image was not only a depiction of a statistical 
norm. It was also invested with the notion of an ideal form. For 
Quetelet, the average man was not only a type but also a sociological 
ideal. Galton’s first composite portrait is indicative of this desire for 
an ideal; it was constructed not from photographs but rather from 
a collection of medallions featuring the head of Alexander the Great 
(figure 14). Galton took six separate medallions carved by different 
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artists and created a composite from photographic images of each 
of them. He explained that each medallion contained individu- 
al faults caused by the artist’s mistakes, but, once composited, all 
these mistakes fell away, leaving a “pure,” idealized representation 
of Alexander the Great. Galton later applied this same desire for an 
ideal to produce composites reflecting norms of intelligence and 
of feminine beauty (the latter made up, exclusively of young white 
women). 

Galton’s most well-known composites were of criminals. Crimi- 
nal portraiture, with its consistent format, provided an ideal source 
for composite production. A composite Galton made of the “violent 
criminal” type (figure 15) provided him with an opportunity for 
deeper reflection on the composite’s ability to produce an ideal. He 
stated: 


It will be observed that the features of composites are much bet- 
ter looking than those of the components. The special villainous 
irregularities in the latter have disappeared and the common hu- 
manity that underlies them has prevailed. They represent not the 
criminal, but the man who is liable to fall into crime.*° 


Galton’s goal was to produce a generalization that would depict the 
features of a violent criminal, but instead he saw an ideal, a face de- 
void of what he understood to be criminal features. Instead, he took 
the composite to depict what a man looks like before he commits 
the violent act. Galton supposed that the irregularities linked with 
deviation from a norm had faded from the image, taking with them 
the deviant characteristics of the criminal. The criminal composite 
projected, rather, a temporal reversal: that is, the face of a criminal 
before the crime. This was an early recognition of the preemptive 
and predicative capacity of the composite, statistical form. 


20 Ibid. 
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Figure 15: Francis Galton, Frontispiece for Havelock Ellis, The Criminal, 
(New York: Scribner & Welford, 1890) 
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The connection between AFR technology and historical practices of 
recognition has been noticed by a few contemporary scholars. In 
this context, mention is often made of the work of Alphonse Bertil- 
lon” and his system of recognition within criminology. For example, 
Kelly Gates relates contemporary AFR systems to Bertillon’s work 
on the basis of an underlying “individualizing logic” that has, as its 


21 Alphonse Bertillon (1853-1914) was a Parisian police official who developed 
an anthropometrically based identification system for tracking recidivism 
in criminals. 
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aim, “to identify individual faces rather than facial types.””” While 
it is true that the eigenface method, along with all contemporary 
automated systems of facial recognition, aims to recognize the in- 
dividual and not the type, I focus rather on the shared logic of rec- 
ognition present in Galton’s composite portraits and the eigenface 
method, a logic based on a reductive and statistical way of seeing. 
The analogy between Galton’s approach and the eigenface method 
is brought home when one sees eigenpictures with captions such as 
“the average man” in journal articles on the method’s development. 
We may also detect an echo of Galton’s process of making a compos- 
ite in Sirovich and Kirby’s description of the process of building a 
training set. Most importantly, the same process of reduction that 
occurs in Galton’s perception of the composite is utilized as a repre- 
sentational mechanism in eigenface. 

As I have explained, Galton’s composite images made it possi- 
ble to perceive the characteristics of an individual to pass into the 
group. Through the recognition process of eigenface and the func- 
tion of the eigenvector, the characteristics of the individual are ex- 
tracted from the group. Both occurrences of recognition rely on a 
composite form; in the eigenface algorithm, this is manifested as 
the eigenvector, which functions as the primary referent for recog- 
nition. Individuals are recognized (or not) based on their deviation 
from an average, that is, the eigenvector. Although eigenface is not 
used to produce and recognize a type, this classifying logic lives on 
as a “ghost in the machine.” Built into the representational mecha- 
nism of eigenface is the ability to recognize individuals in virtue of 
the degree to which they deviate from a norm.” In the example of 
Galton’s composite portraits, part of his broader theory of eugenics, 
we can see the particular cultural and historical context behind the 
merging of statistics and visual perception we find in the logic of 
eigenface. 


22 Gates, Our Biometric Future, 20. 


23 See further discussion on this in Chapter 9: Conclusion, p. 182. 
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Chapter 4: Wittgenstein 
and the Composite Portrait 


Galton’s approach to the composite image has mostly been analyzed in 
relation to social statistics. Yet, as I have mentioned, another signifi- 
cant influence on Galton’s practice of composite portrait production 
was Locke’s theory of abstraction. The composite and its process of 
perceptual emergence is also an object of study within philosophy. The 
philosopher Ludwig Wittgenstein produced a Galton-inspired com- 
posite portrait in 1928. The meaning that he drew from the composite 
image was quite different from the meaning Galton found in it. Like 
Galton, Wittgenstein approached the composite image as an image of 
a thought process. Yet he used the image as a tool in his own investiga- 
tion of language games and the doctrine of “family resemblance” — an 
approach closer to Locke’s study of the formation of words as signs rep- 
resenting general ideas. In his writings, Wittgenstein argued against 
the theory of abstraction, claiming that knowledge that derives from a 
logic of reduction is antithetical to the work of a philosopher. He states: 


Our craving for generality has another main source: our preoccupa- 
tion with the method of science. | mean the method of reducing the 
explanation of natural phenomena to the smallest possible num- 
ber of primitive natural laws: and, in mathematics, of unifying the 
treatment of different topics by using a generalization. This ten- 
dency [...] leads the philosopher into complete darkness.' 


1 Ludwig Wittgenstein, Preliminary Studies for the “Philosophical Investigati- 
ons”: Generally Known as The Blue and Brown Books (Oxford: Basil Blackwell, 
1958), 18. 
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Wittgenstein’s view of the meaning conveyed through the compos- 
ite image is, I argue, opposed to Galton’s, and in this way it runs 
counter to the logic of recognition through statistical means. I 
introduce Wittgenstein in this discussion primarily in order to 
provide an alternative reading of the composite portrait and, con- 
nectedly, to allow for a rethinking of notions of recognition and of 
the relationship between perception and knowledge. I am by no ac- 
count a Wittgenstein scholar, and I recognize that his work is part 
of a wider philosophical canon. But here I want to refer solely to his 
views concerning the composite image and to his implicit critique 
of reductive methods of knowledge production, for these bear on 
the counter discourse I wish to direct against the statistical ways of 
seeing found in facial recognition. 


A Destabilization of Vision 


Wittgenstein often refers to ocular metaphors in describing his 
goal of conceptual and linguistic clarity. In his investigation into 
what he calls language games, he looks toward simplified models of 
language in order to understand the workings and process of lan- 
guage acquisition. The study of language games is in part a way of 
revealing the processes of thought that underlie the use of words, 
what he also referred to as “operating with signs.” Wittgenstein 
was interested in understanding the use of primitive forms of lan- 
guage, such as the ways in which a child begins to use words, and in 
looking closely at the everyday use of language. An important moti- 
vation for Wittgenstein’s investigations was, as he states, “to bring 
words back from their metaphysical to their everyday use.” For 
Wittgenstein the meaning of words does not lie in abstract ideas 
but rather is found through their actual use, which is dynamic and 
mutable. In order to investigate language in the actualities of every- 
day use, Wittgenstein also thought it important to possess a phil- 
osophical ability to “see” the everyday anew. Wittgenstein states: 
“philosophy is an activity which involves relearning how to look.” 


2 Ibid., 16. 
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Through his writing and the inclusion of ocular metaphors (such 
as his well-known reference to the “duck-rabbit”), he advocated a 


destabilization of perception as a trigger for an ability to see anew. 


This reference to reorienting one’s whole visual perspective was 
profoundly connected, for Wittgenstein, to the opening up of new 
ways of thinking and knowing. The composite portrait was one such 
metaphor, and Wittgenstein deployed it in connection to language 


games, specifically with regard to the issue of the unity of a concept. 


Figure 16: The components of the composite photo: Wittgenstein’s sisters 
Gretl, Helene and Hermine, and Ludwig. ©Ludwig Wittgenstein Trust, 
Cambridge 
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Figure 17: Ludwig Wittgenstein’s composite portrait, 1928. 
Photographs by Moritz Nahr. ©Ludwig Wittgenstein Trust, Cambridge 


Drawing on Galton’s experiments with the composite portrait, 
Wittgenstein created his own composite portrait with the help of 
a friend, the photographer Moritz Nahr. Wittgenstein’s composite 
was compiled from photographs of himself and of his three sisters, 
Gretl, Helene and Hermine (figures 16 and 17).? As one can see in 


3 Michael Nedo, of the Wittgenstein Archive in Cambridge, states that the 
negatives used in the making of the composite no longer exist and the exact 
frames cannot be located: “Nahr was exposing negatives of four portraits 
he had taken in a very precise position and with only one background of 
Wittgenstein and his three sisters onto the same photographic paper. Those 
negatives no longer exist but from them and/or other negatives which were 
produced by Nahr in the same context he produced prints some of which 
Wittgenstein pasted into his photo album, and it is those prints we have got 
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the composite image, the four faces are closely aligned, producing 
an impression of a single person’s face in the center of the frame. 
Looking between the four individual portraits and the composite, 
one can detect all the features of the individuals within the compos- 
ite. The composite image and the individual photographs present 
a perceptual enigma; one detects both the individual features and 
their integration into a single face. For Wittgenstein the composite 
functioned as a model of a philosophical method. It illustrates the 
formation of a concept, which he describes through his doctrine of 
family resemblance: 


Consider for example the proceedings that we call “games.” | mean 

board-games, card games, ball games, Olympic games, and so on. 
What is common to them all? [..] if you look at them you will not 
see something that is common to all, but similarities, relationships, 
and a whole series of them at that. To repeat: don’t think, but look! 

[...] many common features drop out and others appear [...] we see a 

complicated network of similarities overlapping and criss-crossing: 
sometimes overall similarities, sometimes similarities of detail. | 

can think of no better expression to characterize these similarities 

than “family resemblances”; for the various resemblances between 

members of a family: build, features, colour of eyes, gait, tempera- 
ment, etc. etc. overlap and criss-cross in the same way. — And I shall 

say: “games” forma family.* 


In this passage, Wittgenstein implores one to look (not to think). In 
this, he demonstrates an aversion to theoretical and idealized pre- 
conceptions, a preference for a pure act of vision over any previous 
knowledge. After imploring us to look, he then describes a (mental) 
image of a “network of similarities” overlapping, such as one finds 
in the resemblances between family members. One could imagine 
Wittgenstein’s composite as such an image. The composite portrait 


and they are close enough to the ones Nahr will have used.” Email message 
to author, February 26, 2019. 

4 Ludwig Wittgenstein, Philosophical Investigations, trans. G. E. M. Anscombe, 
P. M. S. Hacker and J. Schulte, 4th ed. (Chichester: Blackwell, 2009): $66—67. 
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depicts the relationships described as a “criss-crossing” of similari- 
ties. It visualizes not only the commonalities, and thereby the form 
of a concept (such as “games”), but also, as Wittgenstein describes, 
a complex relationship between both similarities and differences, 
allin an instant. 

Wittgenstein mentions the composite explicitly when describ- 
ing the tendencies in philosophical thinking toward a “craving for 
generality.” He describes as mistaken, the belief that a person who 
has understood a term — he gives the example of “leaf” - possesses 
this understanding in virtue of having gone through a visual pro- 
cess of reduction and abstraction from a variety of particular in- 
stances, in this case particular leaves. He states: 


He was shown different leaves when he learnt the meaning of the 
word “leaf”, and showing him the particular leaves was only a me- 
ans to the end of producing “in him” an idea which we imagine to be 
some kind of general image [...] we are inclined to think that the ge- 
neral idea of a leaf is something like a visual image, but one which 
only contains what is common to all leaves. (Galtonian composite 


photograph).° 


Wittgenstein’s reference to the leaf connects his thoughts here to the 
issue of the particularity of organic forms. Leaves, much like fac- 
es, exhibit endless variation. Wittgenstein explains that a general 
term arises out of an act of looking for a commonality. In describing 
the formation of a general concept through the example of the leaf, 
Wittgenstein suggests that generality functions as the end result of 
language. In this there lies a kind of circular logic and a shortcom- 
ing, in that the particularities are all but erased for the purpose of 
naming, that is, providing a generality. Wittgenstein argues that 
philosophers who produce such generalities confuse the sign with 
the object (or objects) to which it points. Wittgenstein describes 
the job of the philosopher as “purely descriptive.” The philosopher, 


5 Wittgenstein, Blue and Brown Books, 17. 
6 Ibid. 
7 Ibid., 18. 
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Wittgenstein says, should not reduce phenomena but describe them 
in all their varying forms. The tendency to adopt scientific methods 
of reduction — the “craving for generality” or, as Wittgenstein also 
puts it, the “contemptuous attitude towards the particular case” — 
runs contrary to this conception of the proper role of the philoso- 
pher. In this, he puts forward an argument against forms of knowl- 
edge that aim to reduce variation through tools of classification and 
the statistical promise of finite calculation. 

In these criticisms, Wittgenstein produces a kind of interven- 
tion — he invites us to engage in a perceptual inquiry into the space 
between the sign and the objects it names. The sign is loosened from 
the grip of fixed ideas. In this space of inquiry, there is a fluidity 
of meaning, possibility and variation. Thus, Wittgenstein’s sense of 
what the composite shows us is very different from Galton’s. Giv- 
en Wittgenstein’s interest in paying attention to particulars, he 
had an altogether different perceptual interest in the composite 
portrait. Wittgenstein describes the composite portrait as “a pic- 
ture of probabilities.”” The composite is an image of multiple per- 
ceptual outcomes rather than a singular probability of a type. The 
composite image has an ability to exhibit all the particular instanc- 
es at once. Because of this ability to show all aspects of a concept 
together, Wittgenstein describes the composite as liberating the 
eye. Instead of focusing on the center of the image as the source of 
meaning, Wittgenstein focuses outside the center of the frame, and 
specifically on the blur in the composite image. This attention to the 
blur inverts the composite’s function of defining a “type,” thereby 
reframing the composite’s utility. It is equally important that one is 
able to perceive all the “particular” cases that spin out of the gener- 
ality, and therefore the generality is not an exclusion of particulars 
but rather an area in which they overlap. Instead of seeing one face 
emerge from beneath the layers of the composite image, Wittgen- 
stein observes the composite’s ability to depict variation and the 
spaces in between faces. In this way, Wittgenstein suggests that 


8 Ibid. 
9 Ibid., 30. 
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there is a kind of perceptual movement present in the composite 
that Galton did not perceive. Galton’s was a static observation. 


Aspect Perception and Aspect Blindness 


An idea of Wittgenstein’s that relates closely to his approach to 
the composite image is that of aspect perception. His descriptions 
of the concept are brief and fragmentary, yet the distinctions he 
makes in the course of its definition suggest a wider importance. 
He describes aspect perception as an ability to perceive one form 
changing into another. He gives the example of perceiving a face: 
“I observe a face, and then suddenly notice its likeness to another. 
I see that it has not changed; and yet I see it differently. I call this 
experience ‘noticing an aspect.” Although he does not directly ref- 
erence the composite portrait in his discussion of aspect perception, 
the reference to recognizing one face’s likeness to another echoes 
his remarks on the doctrine on family resemblance. Wittgenstein 
describes the moment of noticing an aspect as the “lighting up” of 
an aspect," as if it is a sudden occurrence or event. Wittgenstein’s 
moment of “lighting up” is to be distinguished both from some 
combination of looking and thinking and from an act of interpre- 
tation. He describes it as “half visual experience, half thought.” 
Rather than interpretation, which involves forming a hypothesis 
and proving it to be either true or false, aspect perception occurs 
during what he describes as a “state of seeing,” suggesting that he 
is prioritizing the experience of visual sense perception. In this, he 
means to separate seeing from knowing and from being directed 
by language. 

When describing aspect perception, Wittgenstein referenc- 
es Joseph Jastrow’s duck-rabbit illusion. Aspect perception is not 
about an ability to see the duck and the rabbit but rather “the ex- 


10 Wittgenstein, Philosophical Investigations, xi, $113. (Italics in original). 
11 Ibid. 

12 Ibid., 207e, $140. 

13 Ibid., 223e, $250. 
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pression of a change” between the two." Wittgenstein also gives an 
example of an illustration of a box that appears in several places in 
a book, wherein the accompanying captions describe it differently 
each time, providing for different interpretations of the box. One 
sees the box differently each time depending on the words that are 
used to describe it. In contrast, aspect perception is an ability to see 
something differently, to see both or multiple objects as they are, 
that functions not through a shift in thought or words but rather 
through a perceptual ability to see the change and movement from 
one form to another. He states: “what I perceive in the lighting up 
of an aspect is not a property of the object, but an internal relation 
between it and other objects.” Aspect perception requires not only 
a recognition of different forms and concepts but also the ability to 
see the movement and relationships between them. Wittgenstein’s 
concept of aspect perception thus suggests a kind of seeing that al- 
lows one to see things as other. This concept relates to an ability to 
perceive particulars in the composite portrait and a set of relation- 
ships in a doctrine of family resemblances in that there is a value 
placed on an interplay between forms. To better comprehend aspect 
perception and its relevance to my wider analysis, we can refer to its 
opposite: Wittgenstein’s concept of aspect blindness. 

Wittgenstein highlights the importance of the will of the ob- 
server in being able to see difference aspects. Seeing an aspect is a 
voluntary act and, as Wittgenstein puts it, it is “subject to the will.” 
Moreover, he poses the possibility of someone having “aspect blind- 
ness.” Wittgenstein describes the “aspect blind” as those who are 
engaged in a non-dynamic “continuous seeing.”” It is described as 
a failure to “be struck,” as a perceptual state in which one “keeps 
on seeing the same.”* Wittgenstein asks: “Could there be human 
beings lacking the ability to see something as something - and what 


14 Ibid., 222e, $130. Thanks to discussions with Tom Mitchell for pointing 
this out. 


15 Ibid., 223e, $247. 
16 Ibid., 224e, §256. 
17 Ibid., 162. 

18 Ibid., 56e, $129. 
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would that be like? What sort of consequences would it have? [...] 
Is he supposed to be blind to the similarity between two faces? — 
And so also to their identity or approximate identity?” Although 
Wittgenstein does not write very much about aspect blindness, the 
primary feature of this (conceptual) condition seems to be an in- 
ability to see things as otherwise. His brief descriptions of aspect 
blindness strike me as an appropriate description of the perceptu- 
al emergence of a type in Galton’s composite portrait. For Galton, 
clarity in the composite portrait is made possible precisely through 
what Wittgenstein describes as a state of “continuous seeing.” It is 
also precisely this ability to “keep seeing the same” that allows for 
the emergence of a type and, connectedly, that allows for Galton to 
find a perceivable, clear meaning in the composite portrait. And yet, 
for Wittgenstein, this is a form of blindness. For him, the source of 
perceptual clarity in the composite image is to be observed in the 
blur. It is in the zones of indistinction that the dynamic perception 
of aspects is possible. 


Negotiations of Recognition 


What significance does Wittgenstein’s approach to the composite 
portrait and his concepts of aspect perception and aspect blind- 
ness have for the notion of recognition and for the analysis of AFR 
technology generally? Wittgenstein’s work inspires a possible al- 
ternative approach to the perceptual meaning of the eigenface 
image. Wittgenstein’s thought runs counter to the reductive logic 
of statistics that is conveyed by and realized in Galton’s composite 
portraits, and instead he presents an account of the qualities of vi- 
sual perception and their ability to present alternative pathways of 
thought and knowledge production. For Wittgenstein, visual per- 
ception is a dynamic engagement that allows for a constant renewal 
in understanding — something that language and words alone can- 
not do. Taking Wittgenstein’s approach as my point of departure, 
I confront the composite image’s contemporary manifestation, the 


19 Ibid., 224e, $257. 
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eigenface image. The eigenface image may be seen as a moment of 
machinic aspect perception, a moment frozen between the multiple 
probabilities of recognition and the actualization of its operation. 
The eigenface image, expressive of a statistical pattern, is also an 
image of probabilities — depicting a point in the algorithmic process 
of multiple possible outcomes. The blur that constitutes its form is 
constructed by a collapsed archive of faces, faces that are “known” 
by the algorithm. At this stage of aggregation in the algorithmic 

process, the presentation of multiple forms creates, for the human 

observer, an image on which it is impossible to perform an act of 
recognition. It is an image that negates singular recognition. This is 

precisely why it is an image that is expressive of its actual use with- 
in the sociopolitical contexts in which it is implemented. 

I argue that, as a contemporary production of a composite por- 
trait, the eigenface image conveys a certain truth about contempo- 
rary identity. As Btihaj Ajana argues, biometric recognition enacts a 
form of biopolitics, constructing politicized notions of identity that, 
in turn, are constituted by a whole set of practices that govern the 
individual through the hierarchical power relations that not only 
marginalize the vulnerable, such as the asylum seeker, but also em- 
power those on the opposite end of the geopolitical spectrum, the 
neoliberal citizenry with a “surplus of rights.””° In this highly polit- 
icized terrain of mobility, AFR implementation increasingly inter- 
venes to certify identity. And yet notions of identity have never been 
more uncertain. Individuals who are sorted into highly politicized 
categories such as “terrorist” can no longer be classified according to 
fixed social parameters of nationality, ethnicity, racial background 
or even ideology, but rather are increasingly dispersed among the 
broader civilian population. As highlighted during the recent mi- 
gration crisis in the European Union, the abstractions of national 
sovereignty and national status are undermined when Fortress Eu- 
rope is confronted with the mobile, fluid identities of a mass and 
growing population of people defined as “undocumented,” a term 
that signals a loss of all sense of nationhood. The eigenface image, 
in its composite depiction of multiplicity and blur, corresponds to 


20 Ajana, Governing through Biometrics, 5. 
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a contemporary socio-political reality: it captures the transience of 
the subject and the fluidity of identity in mediated form. On this 
alternative reading, the eigenface image’s blurriness is taken to 
convey a malleability of identity. The faces that are displayed in the 
eigenface image resist arrest, resist quantification, that is, “con- 
cretization,” and remain instead dynamic entities. This is not only 
an alternative, more open way of perceiving these images; in light 
of the contexts of AFR implementation and the connection between 
this technology and the construction of contemporary identity, it 
opens up a space for the exercise of the right to self-determination 
— a space that is negated by the operations of automated facial rec- 
ognition. 


Concluding Remarks 


In this analysis, I have shifted attention toward the aesthetics of the 
eigenface image as a way of investigating the notion of recognition 
that is in play in the operation of an AFR system. In relating the 
production of these images to composite portraiture, I have sought 
to problematize the underlying logic of recognition, which is based 
on a statistical way of seeing. This socio-historical analysis began 
by tracing this statistical way of seeing and logic of recognition in 
two images, the eigenface image and Galton’s composite portrait. 
Wittgenstein provides an alternative take on the composite portrait 
and sees an alternative logic in the image. Wittgenstein’s approach 
reveals how the composite form, which Galton had endowed with 
statistical relevance, captures a way of seeing that is dynamic, fluid 
and probabilistic. For Galton, the composite portrait was part of a 
larger project, his theory of eugenics. For Wittgenstein, the com- 
posite portrait was connected to his wider project of a philosoph- 
ical investigation into language and his doctrine of family resem- 
blance. These approaches contrast with each other on many levels 
(including in terms of Galton’s and Wittgenstein’s different fields of 
study), yet the structure of the composite form is the same in both 
cases, and it is this same form that structures the eigenface im- 
age. Through these different approaches, the composite form has 
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emerged as symbolic of the paradoxes that are inherent in a facial 
recognition process. 

Wittgenstein’s view of the perceptual intelligibility of the com- 
posite portrait and his concept of aspect perception together suggest 
a re-reading of the eigenface image. On the Wittgensteinian view, it 
is an image that conveys a resistance to categorization according to 
the common denominator, a resistance that allows all particulars to 
remain; the individual does not disappear. Wittgenstein sets out a 
critique of the logic of reduction that structures statistical recogni- 
tion, providing alternative conceptual lenses through which to view 
the operation of machinic vision. The inclusion of his critique in this 
analysis opens up a space for a perceptual clarity that avoids the 
algorithm’s reductive procedures of statistical recognition. I would 
like to concur with Wittgenstein in calling for a perceptual intelligi- 
bility and a destabilization of vision as I now turn to contemporary 
art production as it engages with practices of facial recognition — as 
a source of revelation, recontextualization and reimagining. 


Part Il: Artistic Interventions 
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The second part of this study continues to engage in a cultural 
analysis of AFR technology, but it does so by focusing specifically 
on the work of contemporary artists. Bringing the overall analysis 
into the present, this part explores the ways in which contemporary 
artworks engage with the topic of facial recognition, both revealing 
and reimagining the cultural dynamics behind its technology and 
forms of representation. I have chosen three artists, Thomas Ruff, 
Zach Blas and Trevor Paglen, whose works contribute to a critical 
discourse on facial recognition and whose work I have chosen spe- 
cifically on the basis of its engagement with the socio-political con- 
texts and implications of facial recognition practices. Their works 
shed light on the role of art as a source of theoretical reflection, and 
they engage experimentally with the technology of facial recog- 
nition. Each builds on some of the critical work of contemporary 
cultural studies scholars mentioned in previous sections, providing 
nuanced visual articulations of these scholars’ contributions. 

In the context of the present study, the work of these three art- 
ists supplements the foregoing analysis of the eigenface algorithm 
(with one artist, Paglen, directly referencing this method in his 
work). Each artist presents an artistic rendering of the composite 
form, an extended analysis of the role of the composite form and its 
relationship to facial recognition. These artistic articulations of the 
composite echo Wittgenstein’s sentiments with regard to compos- 
ite images: they reject a reductive reading of the images and instead 
open up a space for perceptual fluidity. Although all three artists ar- 
ticulate critical perspectives on facial recognition, their approaches 
to the topic contrast greatly with one another. They each approach 
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the topic through a variety of mediums, a fact that itself reflects 
the range of meanings attributable to facial recognition and pos- 
sible responses to this phenomenon. Together they provide a wide 
spectrum of views on facial recognition and its use in society today. 

Growing numbers of artists are working with facial recognition 
technologies and engaging, more broadly, with the sociopolitical 
issues surrounding surveillance. Thanks to imagery that allows for 
a more nuanced and open-ended approach to a topic, artistic en- 
gagements with facial recognition technology are able to articulate 
some of the more complex issues that arise in this area. As the de- 
velopment of AFR systems continues to be, for the most part, guid- 
ed by the sorts of contexts in which these systems are used, such as 
policing, the military and marketing, artists engaging with facial 
recognition technology appropriate the technology and remove it 
from these original contexts; in doing so, they not only highlight 
the technology’s political and social implications but also imagine 
alternatives to the use of AFR processes and production. 

Art production, being rooted in the relationships of the visual, 
is in a unique position to articulate the underlying questions of this 
analysis, which have to do, again, with the forms of visuality on 
which AFR systems are based and the transformations in cultural 
meaning associated with their operation. Although the eigenface 
method produces an image in the course of its operation, most other 
AFR methods do not. AFR systems and the other forms of machinic 
vision used in networked surveillance systems often operate in a 
clandestine fashion and, as such, result in a kind of aesthetic de- 
fined by absence, that is, by a lack of images and visual information. 
For this reason, the artworks that engage with these technologies 
produce a visibility where there previously was none. Artists such 
as James Bridle, Omer Fast and Hito Steyerl and the collaborative 
group Forensic Architecture have addressed these issues in relation 
to surveillance technologies more broadly.' These artists provide a 


1 See James Bridle’s “Drone Shadow” series, images of which are available 
at “Drone,” James Bridle website, accessed June 3, 2019, https://jamesbridle. 
com/works/category:drone; Omer Fast’s short film 5,000 Feet is the Best 
(2011), digital video, http://www.gbagency.fr/en/42/Omer-Fast/#!/5-000- 
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kind of “counter imagery”; that is, they produce images that count- 
er a lack thereof. Their work is imbued with a form of political and 
social agency that is enacted through this visibility. Artists who 
address this lack of visibility in surveillance technologies use var- 
ious strategies to do so, including playing with the visual syntaxes 
of documentary images, merging the physical world with imagined 
worlds constructed through virtual imagery. These artistic engage- 
ments with contemporary surveillance technologies provide a cru- 
cial component that is missing from the academic cultural analysis 
of AFR and surveillance technologies. They provide images that 
both problematize and reimagine the use of these technologies. In 
addition, many artistic works contrast with scholarly cultural anal- 
yses of the technology in that they leave the implications and out- 
comes of these issues open ended and in flux. 

At the intersection of artistic intervention and AFR technology 
is an ability to make visible what may be latent in the technology. 
As well as an operation of recognition, AFR systems involve an op- 
eration of representation. The relationship between these two is 
unsettled. As I explained in the previous chapter on eigenface, the 
algorithms used in AFR systems require facial models that are con- 
structed on the basis of training data. These virtual models func- 
tion as a source of face knowledge for an AFR system. These systems 
rely on representational mechanisms that structure the knowledge 
acquisition of the algorithm. These representational mechanisms 
are related to a discourse to do with the representation of faces in 
the art historical genre of portraiture. Art curators have noticed 
this relationship. Curator Cornelia Kemp argues that the facial rep- 
resentations found in AFR processes are particularly connected to a 
cultural history of photographic portraiture. She explains, 


Feet-is-the-Best/site_video_listes/88; Hito Steyerl’s video How Not to Be 
Seen: A Fucking Didactic Educational .MOV file (2013), digital video, https:// 
www.artforum.com/video/hito-steyerl-how-not-to-be-seen-a-fucking-di- 
dactic-educational-mov-file-2013-51651; and Forensic Architecture’s video 
Drone Strike in Mir Ali (2013), digital video, https://forensic-architecture. 
org/investigation/drone-strike-in-mir-ali. 
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researchers of facial identification have produced a veritable ar- 
senal of alienated photographs using superimposition, collage, 
reduction, and abstraction in an effort to divulge the secrets of hu- 
man face perception. The astonishing findings [...] lead back to the 
metamorphoses of the portrait in art.” 


Researchers and developers reference methods of art historical 
practices of portraiture. In addition, artists reference scientific 
practices of representing the body. Curator Louise Wolthers de- 
scribes a strategy, connected to photographic portraiture, adopted 
by contemporary artists engaging with AFR technology when she 
states: 


The reduction of the unruly, material body to an unambiguous 
representation is a practice in which analogue photography has 
played a major role, and in which new digital — and camera-less — 
technologies perpetuate. Works by artists [...] emphasize this by 
rematerializing or disrupting this form of representation.’ 


Artistic engagements with AFR technology continue a metamor- 
phoses of forms of portraiture practice in the digital age, while also 
examining how these shifting representations of the face relate to 
contemporary notions of identity. 


Phantom Portraits 


Before I enter into an analysis of specific artistic engagements with 
AFR, I would first like to discuss more broadly the genre of portrai- 
ture and its relationship to notions of representation and recogni- 


2 Cornelia Kemp, “The Other Face,” The Other Face: Metamorphoses of the Photo- 
graphic Portrait, ed. Cornelia Kemp and Susanne Witzgall (Munich: Prestel 
Verlag, 2002), 13. [8-13] 

3 Louise Wolthers, “Introduction: Watching Europe and Beyond - Surveil- 
lance, Art and Photography in the New Millennium,” in Watched! Surveil- 
lance, Art and Photography, eds. Louise Wolthers, Dragana Vujanovic, Niclas 
Östlind, (Köln: Walther König, 2016), 18. 
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tion. Portraiture as a genre in art history involves a representative 
depiction of a person and focuses on the face as the central theme. 
Historically, the subject was often a person with an important so- 
cial status, and the depiction captured an expression of the subject’s 
character, societal context and internal emotional life. With the 
advent of photography and the introduction of mechanical repro- 
duction processes, other genres of portrait, other than the artistic 
portrait, were born, including the identification portrait produced 
within the context of institutional bureaucracy and policing. In- 
stead of depicting the character of a subject, the identification por- 
trait focuses on the measurable features of a subject’s face. Portrai- 
ture in the age of machinic vision has made possible still further 
categories of the genre. Automated facial recognition creates new 
ways of reading the face, exploiting perspectives that, because of 
their scale, are inaccessible to human recognition. These potential 
new forms of portraiture, that is, new forms of facial representa- 
tion, can be understood as responses to the technology, to the act of 
recognition, or as produced by the technology itself. 

For example, the artists Adam Broomberg and Oliver Chanarin 
have termed one such new form of portraiture the “non-collabo- 
rative portrait”: that is, the production of a facial image of a sub- 
ject who is “neither consensual nor necessarily aware of the cam- 
era.”4 Broomberg and Chanarin explain that this term originated 
from discussions with engineers who used the phrase to describe 
the challenges they faced in designing AFR technology. This term 
brings to the fore issues of privacy, surveillance and consent in re- 
lation to the everyday visibility of the face as a potential site of tres- 
pass. Broomberg and Chanarin use this term as the title of a series 
of portraits they produced using a facial recognition system that 
creates a kind of three-dimensional mapping of faces. In this se- 
ries of portraits, the subject’s gaze is directed off into the distance. 
They state: “there is never a moment in the capturing of the ‘image’ 
when human contact is registered; the subject’s gaze, or any con- 
nection between photographer and sitter that we would ordinarily 


4 Adam Broomberg and Oliver Chanarin, “The Bone Cannot Lie,” in Adam 
Broomberg and Oliver Chanarin, Spirit is a Bone (London: Mack, 2015), 207. 
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rely on in looking at a portrait, is a complete fiction in this space.”5 
In appropriating this term to provide a conceptual framework for 
their artistic research, Broomberg and Chanarin underline a cen- 
tral aspect of the visuality of AFR technology as it contrasts with the 
more stable understanding of the genre of portraiture: that is, the 
absence of a visual reciprocity.® 

Algorithmic artifacts such as the eigenface image analyzed 
in the previous part of this study present yet another new kind of 
“portraiture” in the age of AFR. Unlike Broomberg and Chanarin’s 
“non-collaborative portrait,” the eigenface method is dependent on 
the forward-facing pose and neutral expression of its subject, and 
the consent of the subject is obtained. Present in the eigenface im- 
age is a history of representations of the face in photographic iden- 
tification portraiture. What is different, and apparent, in the eigen- 
face portrait is the presence of the perspective of the machine. In 
2003, the artist and filmmaker Harun Farocki used the term “phan- 
tom images” in an essay of the same title. The phrase had been used, 
he says, to describe cinematic shots from the 1920s, taken from a 
perspective “that a human cannot normally occupy.” Farocki gives 
the example of a movie camera attached underneath a train, giv- 
ing the viewer an ability to experience a vantage point that a hu- 
man could not physically occupy. The phantom perspective can be 
described not only as lying outside the limits of human physicality 
but also as beyond the scale of human visual sense. With Farocki’s 
concept of phantom images in mind, I propose that another cate- 
gory of portraiture has emerged in the age of AFR technology: the 
“phantom portrait.” The phantom portrait is the result of the bio- 
metric reading of faces, perceived from the perspective of a disem- 


5 Ibid. 

6 This lack of visual reciprocity may been seen in earlier artistic photographic 
portraiture, for example in Helen Levitt’s historical subway photography. 
See Helen Levitt, Manhattan Transit: The Subway Photographs of Helen Levitt 
(Köln: Walther König, 2017). Yet, Levitt‘s photographs depicted an emotive 
character of her subjects. For Broomberg and Chanarin, this lack of visual 
reciprocity is a defining element within the contexts of facial recognition 


and surveillance. 
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bodied, machinic eye. Although it is characterized by a non-human 
perspective, the phantom portrait depicts the human form of the 
face and presents the visualization of a mathematical abstraction. 
As part of a reexamination of the genre of portraiture in relation to 
automated facial recognition, the notion of the phantom portrait 
raises questions of representation and of the relationship between 
this technology and notions of identity. 

The phantom portrait can be understood as a “counter image,” 
specific in its response to AFR technology, providing visualization 
of the contexts and implications of its use that would otherwise 
be non-existent in regards to the output of its actual processes. It 
makes visible what is latent in AFR technology: ambiguous forms 
of facial representation that fall within the cracks of its operation, 
taking into account the “unruly materiality” of the body and dis- 
rupting the standardized procedures of facial recognition. Artists 
such as Bang Geul Han, Heather Dewey-Hagborg and Kristoffer 
Ørum continue to reformulate the genre of portraiture through the 
use of varied technological recognition techniques other than AFR, 
such as facial expression recognition systems, DNA phenotypes 
and facial tracking systems, respectively.® The artists I will discuss 
in the following section, Ruff, Blas and Paglen, reference the per- 
spective of the machine that reads faces, a perspective specific to 
facial recognition practices. One of these artists, Ruff, does not 
directly reference AFR technology, but his explorations of identi- 
fication portraiture are pertinent to a wider analysis of notions of 
recognition and representation. His work also bridges the histori- 
cal discourse of representation in photography with forms of repre- 
sentation by the technology of AFR systems. These artists produce 
forms of portraiture that disrupt traditional representations of the 
face and, as a result, articulate various connected issues concern- 
ing the politics of representation that are inherent to the contexts in 
which facial recognition systems are applied. 


8 See Bang Geul Han, Referential Gaze (2015), inkjet prints; Heather De- 
wey-Hagborg, Stranger Visions (2012-2013), sculpture; and Kristoffer 
Ørum, Ambivalent Physiognomy (University of Copenhagen, May 22, 2017), 
performance. 
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Background: The Composite Portrait in Contemporary Art 


Ihave already discussed the composite portrait as an object of study 
within the fields of science and philosophy. Yet the composite por- 
trait has also been an object of study within the field of art. Instead 
of using it to construct sociological or criminological taxonomies, 
artists have usually approached the composite portrait as a visual 
construct, problematizing the relationship between mechanical or 
digital reproduction and indexicality. One of the most well-known 
artists to work with composite portraits is Nancy Burson, who, in 
the 1980s, produced a series of works that featured the composite 
image and exploited its ability to produce ideals and types. Burson 
curated the categories of her composite portraits, constructing dig- 
itized averages of faces that portrayed sociological ideals of gender, 
power and race. These works expressed the critical role of identi- 
ty politics in defining the sociological ideals. For example, Burson 
constructed a composite portrait, Beauty (1982), which consisted of 
the faces of famous Hollywood actresses, resulting in a depiction of 
a feminine and racialized ideal of beauty. Another work, Warhead 
(1982), is a composite of the faces of the global leaders who were, at 
the time, antagonists in a Cold War standoff, depicting an ideal ofa 
political power defined by the white male. One reading of Burson’s 
composites is that they invert the notion of the ideal that Galton had 
pursued through his composites, instead revealing the racialized 
and gendered parameters within which concepts such as beauty 
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and power are imagined in our society. In this way, Burson is able 
to use the symbolic aspect of the composite in a critique of society. 

Burson’s composite portraits are considered some of the ear- 
liest computer-generated artworks, and aesthetically they exhibit 
a digital seamlessness in which the visualized average can be ob- 
served in a singular yet pixelated face. Thomas Ruff produced a 
less well-known series of composite portraits in the mid-1990s, the 
andere Portraits, utilizing analogue photography and silk-screening 
processes to highlight the disparities between the composited fac- 
es. Rather than focusing on the composite’s ability to produce and 
depict a monolithic aggregate, Ruff’s composites call our attention 
to a constant movement between faces and thus make the zones of 
indistinction obvious. 


Identification Portraiture 


Ruff’s andere Portraits grew out of his earlier series Portrait, an in- 
vestigation of identification portraiture. In the late 1980s, Ruff be- 
gan producing portraits of his colleagues at art school in the style 
of ID card and passport photos. These images grew into a study of 
identification portraiture, the kinds of photographs that function 
to identify individuals based on a clear, forward-facing pose and a 
neutral facial expression that accentuates the measurable features 
of their faces. These portraits explicitly exhibited photography’s 
ability to show only the surface appearance of the subject; rather 
than revealing something about the person, as the viewer would 
expect from an artistic portrait, they left out the context, character 
and personality of the individual. These portraits exaggerated this 
capacity of the photograph to depict only what lies on the surface. 
They discouraged any internal reading of the photograph’s subjects. 

I argue that Ruff and Wittgenstein share a similar approach to 
the investigation of their respective mediums, photography and 
language, both of which are ways of operating with signs. As I men- 
tioned previously, Wittgenstein said that in his investigation he was 
aiming to “bring words back from their metaphysical to their ev- 
eryday use,” that is, to understand language through a study of its 
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actual and everyday use. I argue that this approach is similar to how 
Ruff approaches photography and the photographic image. His in- 
vestigations with photographic images attend very much to their 
everyday and bureaucratic uses, extracting the photographic image 
from a discourse on indexicality and instead approaching photo- 
graphs through their use and as a kind of visual construction. Ruff 
states that photography “can only reproduce the surface of things.”! 
Far from understanding this as a limitation of photography, Ruff 
uses his work to highlight this aspect of the medium. This, for me, 
also harks back to Wittgenstein’s approach to language; he cautions 
against the view that, as he puts it, “the incomparable essence of 
language [...] was hidden from us beneath the surface.” Instead, 
Wittgenstein proposes to liberate us from this urge by getting us to 
recognize that everything “already lies open to view.”” 

Ruff’s work with identification portraits conveys a sense of the 
chasm between an observer of the image identifying with the sub- 
ject’s face and identification as a practice of governance and insti- 
tutional regulation. Ruff’s photographic work is an investigation 
of the utilitarian uses of photography, or, you could say, of how 
photography is used in the everyday. His identification portraits 
concern types of photographs that, in some way, otherwise go un- 
noticed; they are both ubiquitous and unquestioned, utilized for 
specific and usually bureaucratic purposes. What is on display in 
Ruff’s portraits is not only the subject’s face but the conditions of 
its representation. In this, Ruff takes the visual language of the ar- 
chive to represent the individual. Placed in a purely visual realm of 
art, Ruff’s series of identification portraits problematizes the au- 
thority the archive has in relation to the subject of the portrait. As 
Dan Adler puts it: 


1 Richard Dorment, “Photography in Focus,” The Telegraph, May 29, 2003, 
http://www.telegraph.co.uk/culture/3595514/PHOTOGRAPHY-IN-FOCUS- 
The-deadpan-images-created-by-Thomas-Ruff-of-nameless-individuals- 
and-equally-anonymous-places-are-masterpieces-of-austere-neutrality. 
-By-Richard-Dorment-Now-for-something-completely-indifferent.html 
and Hilde Van Gelder and Helen W. Westgeest, Photography Theory in Histo- 
rical Perspective (Hoboken: Wiley-Blackwell, 2011), 43-44. 
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Everything and everyone that enters the archive is said to be sub- 
ject to the same ritualized process of framing, a process that crea- 
tes (and maintains) order, that suggests systematicity but refrains 
from signification. [..] Ruff [..] rigorously and repetitiously enacts 
photographic technologies, with each body of work implying con- 
text of regulatory ordering and categorization, an operation that 
must operate by both including and excluding subjects.? 


One of the results of Ruff’s reenactment of the archival ritual of 
identification photography is the conversion of his photographic 
subjects into objects, as Adler’s reference to refraining from signifi- 
cation suggests. Ruff has stated that, once photographed, the per- 
son is “thingified.” The exclusion of a subjective stance in the image 
occurs through the observation of the image; the photographed 
subject lacks the character that is usually expected of a portrait in 
an artistic context. Instead, Ruff’s portraits evoke the role of pho- 
tography in science, as a way of documenting truth, and when ap- 
plied to the human subject the result is a tension between a truth of 
the subject and the truth of the image. 

Ruff’s Portrait series visualizes what the surveillance scholar 
Lyon describes using his concept of a “data double,” which is a per- 
son’s identity within a database, an identity created from coded cat- 
egories that produce a kind of “virtual fiction.” These identities cir- 
culate amongst multiple institutions and interests and, all the while, 
are “vulnerable to alteration, additions, merging and loss as they 
travel.” The production of a “data double” results in a gap between 
a sense of self and a “data image.” Ruff has described his portraits 
as a kind of “fiction,” and his portrait work has been criticized as 
producing an experience of alienation rather than identification in 
the viewer. Yet this experience is what we might expect of a subject 
confronting their own “data double” — an experience of dissonance 


3 Dan Adler, “The Apparatus: On the Photography of Thomas Ruff,” Art Jour- 
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produced by the representation of themselves as information. Lyon 
describes the “data image” as a digital persona that is created on 
the basis of risk assessment and then circulated and as an exam- 
ple of how surveillance communication “is increasingly distanced 
from the person from whom the data is initially obtained.” Lyon 
explains: “The data-image may be abstract but it is not innocent.” 
In the way that they bring to the surface the glaring bifurcation of 
the observer’s sense of identity through the photographed subject, 
Ruff’s Portraits may be understood as illuminating the aspects of 
data images described by Lyon. Ruff’s study of identification por- 
traits reveals that these forms of representation are not neutral in 
much the same way that, as Lyon explains, data images are not in- 
nocent. 


andere Portraits 


The composites in Ruff’s andere Portraits (1994-1995) were construct- 
ed from his earlier portraiture series. These portraits extend the 
idea of a photographic “fiction” underlying the image. He has ex- 
plained that, with these portraits, he intended to construct faces: 


The original idea was to reconstruct one of my portraits from se- 
veral faces. But | gave up the idea quite soon, it was impossible. | 
developed the idea of “building” new faces, faces that do not exist. 
| mixed — man and man — man and woman — woman with woman 
and woman with men. They should be believable but at the same 
time the viewer should realize they are fictional.® 


Ruff produced these composites by appropriating a security cam- 
era apparatus called the Minolta Montage Unit, used by the State 
Bureaus of Investigation in Berlin in the 1970s. This camera was 
originally used to produce pictures that combined individual facial 
features from four photographs, based on an eyewitness account of 


7 Lyon, “Facing the Future,” 173. 


8 Thomas Ruff, email message to author, May 15, 2016. 
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acrime. The decision about which faces to include in the composite 
was made on the basis of the eyewitness account. Instead of having 
an artist sketch the face, the unit could composite different facial 
features to create a photographic “sketch” of a criminal’s face. The 
resulting composites were often used as mugshots and displayed on 
“Wanted” posters in public spaces. 

The Minolta Montage Unit used four passport-sized photos and, 
by way of an optical mirror system, merged them into a compos- 
ite. There is a built-in polarizing filter to eliminate the hard edges 
where the lines of the facial features do not match up. The Minolta 
unit is a more sophisticated version of Galton’s composite practice 
in the sense that the unit itself mechanizes the process of layering 
the faces. Instead of using four photos, Ruff combined two faces 
taken from his Portraits series. In producing these binary compos- 
ites, Ruff played and experimented with the gender binary, com- 
bining faces of men and women. He then took a photograph of the 
resulting composite and used the photographic slide as the basis 
for a black-and-white silkscreen. He explains: “I wanted to go closer 
to the police technique, that’s why I decided to print them as silk 
screens with the technical structure of a printed medium.” In con- 
trast to the photographs from which they are constructed, which 
were originally exhibited by Ruff in vivid, saturated color, the re- 
sulting images are black and white. For the andere Portraits, Ruff 
consciously chose to avoid color photography, noting that a color 
photograph is much closer to a depiction of how reality looks to the 
unaided eye. 

In his descriptions of this technique, Ruff often refers to the 
security and policing context in which the Minolta Montage Unit 
was originally used. His approach to investigating photography 
and photographic practices often plays with the notion of the pho- 
tograph as a fact and/or fiction, questioning and problematizing its 
evidentiary quality. The use of the Minolta Montage Unit to produce 
mugshots from eyewitness accounts of crimes seems also to merge 
these notions, revealing the resulting composites to be fictional 
constructs that make use of the medium of photography. The “other” 


9 Ibid. 
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referred to in the title of the series — andere meaning “other” in Ger- 
man - can refer to the original use of the apparatus by security bu- 
reaus: Other, with a capital “O,” denoting those who stand outside of 
society, the criminals or the terrorists, those outside of the realm of 
the familiar and deemed dangerous. It can also refer to the face that 
is created when the two portraits are used to make the composite. 
As with Galton’s composites, the fact that this face has no indexical 
referent and exists only in a pictorial reality is highlighted. Where 
the portraits combine the faces of men and women, the “other” can 
also refer to that crossing of a culturally constructed gender bina- 
ry, a crossing that creates the face of an “other” gender that fluidly 
moves between the two traditional gender categories. 


Metaportraits and the Binary Face 


Referring to the reproduction technique used in Ruff’s Portraits se- 
ries, that is, the act of producing portraits of portraits, art histori- 
an Patricia Drück describes these pieces as “meta portraits.” This 
term can equally be applied to the andere Portraits, in that they are 
portraits of an already existing form of portraiture, the composite 
mugshot, and they appropriate the Minolta Montage Unit appara- 
tus in their production. I think Drück’s term is helpful in under- 
standing these images and Ruff’s motivations, because it captures 
the way in which Ruff is producing portraits of a portraiture prac- 
tice in such a way as to reveal the practice’s character and essence. 
In doing this, Ruff presents the resulting composite portrait as an 
object of visual contemplation. 

Ruff’s decision to use only two identification portraits leads to 
a particular sort of binary composite. As Ruff states in the passage 
quoted above, he experiments with gender, combining the faces of 
both men and women. In this way, one might say that Ruff actual- 
ly utilizes some notion of a “type,” in gendered norms, and creates 


10 Patricia Driick, Das Bild des Menschen in der Fotografie: Die Porträts von Tho- 
mas Ruff (Dietrich Reimer, 2004), 170, quoted in Gelder and Wehtgeest, 
Photography Theory, 43. 
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composites of faces that disrupt these binary gender norms. In the 
piece titled anderes Portrait, Nr. 71/65 (1994/95) (figure 18), an incon- 
gruity between gendered norms is made especially visible. At the 
center of the image there is a single face. This face has delicate, fem- 
inine features, with rouged lips. Panning out from the center, two 
separate and incongruent hairlines begin to form. One face appears 
thinner, with a short, masculine hairstyle. The other face, behind 
it, creates a kind of halo effect, encircling the other. A wild mane 
of hair surrounds both faces. From the neck down, there is a single 


black turtleneck. 


Figure 18: Thomas Ruff, anderes Portrait, Nr. 71/65 (1994/95). 
Image courtesy of the artist 
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Another piece, anderes Portrait, Nr. 102/13 (1994/95) (figure 19), pro- 
duces an experience of constant double vision as one observes the 
appearance of the face. Although, like the previous image, there is 
a kind of alignment of faces at the center of the image, the features 
are less sharp, making the face appear to be in constant movement. 
The outlines of the two faces are closer to each other than in the 
previous image, and yet they do not align. The result of this lends 
the portrait a stereoscopic effect, giving the eye of the observer no 
outline to rest on as that of a singular face. The face at the center of 
the image appears feminine because of the obvious makeup, main- 
ly eyeliner, eyeshadow and lipstick, as well as cosmetically shaped 
eyebrows. Yet the overall form and shape of the composite face pro- 
duces an extremely masculine effect; it is stocky and muscular. The 
hairline is starkly defined by a dark hair color and parted to the side. 
As in all the andere Portraits, the facial appearances meld into a sin- 
gular image below the face. In this portrait, the subject has a broad 
neck with a very visible Adam’s apple, and shoulders clothed in a 
man’s button-down shirt. This and the previous image are exam- 
ples of how Ruff’s composites highlight the incongruities between 
the two portraits that make up his composites. Although a single 
face appears at the center of the images, the misaligned outlines of 
the underlying faces immediately unsettles the perception of any 
singular face. A catalogue that was produced for an exhibition of 
the andere Portraits at the Venice Biennale in 1995 reveals that Ruff 
produced a few of these composites in stereo, using the same face 
to produce two different composites." In the catalogue, two sepa- 
rate composites using one of the same faces sit side by side on op- 
posing pages. This adds another level of stereoscopic vision for the 
observer; one can see the same face, but differently. The portrait 
I just mentioned, anderes Portrait, Nr. 102/13 (1994/95), has a kind of 
doppelganger in the piece titled anderes Portrait, Nr. 102/125 (1994/95) 
(figure 20), which uses one of the faces used in the former portrait 
but combines it with a different face. When placed side by side, one 
is able to observe the similarities between these two portraits, most 
significantly in the eyes. But the image gives way, once one turns 


11 Thomas Ruff, Andere Porträts + 3D. (Germany: Cantz Verlag, 1995), 34, 35. 
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to the rest of the face, to the merging and disparity produced by 
the imposition of the “other face.” The shapes of the heads are dif- 
ferent, and the clothing worn, which can be seen clearly from the 
neck down, differs as well. In looking at these portraits side by side, 
some of the characteristics of the shared face emerge through the 
comparison, while, at the same time, the binary structure of these 
portraits is accentuated. 


Figure 19: Thomas Ruff, anderes Portrait, Nr. 102/13, 1994-95. 
Images courtesy of the artist 


Ruff’s andere Portraits consist of overtly and visibly manipulated im- 
ages. Aside from the many layers that are produced by the repro- 
duction practice used to create the andere Portraits, there are also 
many layers within the content of the images. Ruff’s intervention 
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in producing these images is made visible through the use of an- 
alogue mediums, both the photographic apparatus of the Minolta 
Montage Unit and the process of silk-screening. This intervention 
is made more visible to the viewer in the choice of a binary image 
structure, because the viewer sees more clearly the outlines of each 
individual face. There is a seemingly constant movement between 
faces, a binary face that never stays put. 


Figure 20: Thomas Ruff, anderes Portrait, Nr. 102/125, 1994-95. 
Images courtesy of the artist 


Rather than applying a statistical technique that would merge the 
faces into a conglomerate, such that the multiplicity of outlines cre- 
ated a blur, Ruff chooses to use just two images and thus makes the 
misalignment all the more visible. The spaces in between the two 
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opposing or incongruent faces become spaces of uncertainty. In the 
central areas where the two faces align, a third face emerges. This 
face, a “fiction” as Ruff calls it, is apparent and seen in the photo- 
graph, and yet it exists only in the pictorial realm.” Historically, in 
Galton’s use of the composite portrait, this “fiction” was endowed 
with a level of certainty and statistical salience. Yet, in Ruff’s por- 
traits, this “fictional” or “other” face refuses to stand still. In place of 
certainty, one sees double and then, again, multiple faces. The areas 
of commonality are unhinged, uncoupled from their surroundings, 
and produce instead an uncertainty. Once you see a face, it chang- 
es and you see another, and then it changes again. Ruff’s portraits 
produce a constant mutability and expose a process of perceptual 
construction at work in the observer. In this way, the experience of 
the observer of the image is a part of what is on display in the image. 
The technical attributes of the Minolta Montage Unit are de- 
scribed in a trade magazine: “Interestingly, the synthesizer is not 
hindered by racial or sex barriers. It can be adjusted for skin tone 
and texture, allowing any feature mix of male or female.”” This is 
framed as a kind of technological advantage: that the apparatus is 
neutral and unhindered by social norms. Ruff utilizes this feature 
to produce portraits that mutate fixed, binary notions of gender. 
Ruff’s constructions of what he calls “fictional faces” present fac- 
es that reveal not only the negotiations of a technically produced 
representation but also the relationship of the representations to 
the cultural contexts in which identity is negotiated. The andere 
Portraits display the malleability and ambivalence of identity. Dan 
Adler points out that the andere Portraits “demonstrate how the mis- 
use of archival technologies may be read as gestures of a progres- 
sive political position, one that is open to the concept of exposing 
(or at least acknowledging) subjects that cannot be (or resist being) 
classified.”“ Ruff’s andere Portraits reference the relationships be- 
tween surveillance by security forces, archival documentation and 


12 Thomas Ruff, email message to author, May 15, 2016. 

13 Clemens Mitscher, “1987 - Opfer,” Clemens Mitscher website, accessed 
May 25, 2017, http://mitscher.de/content/?page_id=265. 

14 Adler, “The Apparatus,” 7. 
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identity. The use of the binaries that visually structure Ruff’s com- 
posites may also be understood as a reference to the binaries that 
problematize the visual operations of recognition, that is, the inner 
versus the outer, the objective versus the subjective and, as Ruff 
says, “A truth versus fiction.” * 


Scale 


One of the aspects that characterizes Ruff’s work with portraiture 
is the monumental scale on which he has exhibited some of his 
portraits. Combining the photographic portrait with the genre of 
large-scale, life-size painting, Ruff emphasizes the iconic status of 
the photograph and, as Enwezor argues, “emancipates the photo- 
graph from being read as a document, moving it instead towards 
being perceived optically and approached haptically as a picture.” 
The large-scale format for the exhibition of prints was also used by 
Ruff in his exhibition of the andere Portraits at the Venice Biennale 
in 1995. This again affected the way in which these images conveyed 
the meaning of a certain photographic genre of portraiture, in this 
case, the mugshot, by removing them from the contexts of identi- 
fication and policing, and instead allowing observers to approach 
them as pictorial objects of study. 

Although they were not exhibited on a monumental scale, the 
andere Portraits were printed life size so that a viewer, upon con- 
fronting them up close, could witness a breakdown in their tech- 
nical structure through the enlargement of the silk-screened, pho- 
tographic grain (figure 21). Distance is accentuated through scale. 
The size of the portraits makes it so that the viewer has to stand 


15 Ruff, Andere Portrats + 3D, 16. 

16 Okwui Enwezor, “The Conditions of Spectrality and Spectatorship in 
Thomas Ruff’s Photographs,” in Thomas Ruff: Works 1979-2011 (Miinchen: 
Schirmer Mosel, 2012), 11. [9-19] The monumental scale on which his series 
of identification portraits was exhibited in Düsseldorf, Germany drew cri- 
ticism at the time, as it conjured up a cultural memory of a Nazi aesthetic. 
Thanks to artist Johan Réing, who attended the Düsseldorf school with 
Thomas Ruff and pointed this out. 
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at a distance to properly perceive the entire image - to see it as a 
complete portrait. The closer one gets to the image, the more ab- 
stract the image becomes, further alienating the observer from any 
identification with the face. The further away one gets, the more the 
movement between the binary faces is made visible. Rather than 
depicting a clear image of the constructed face of a criminal, as in 
the original composited mugshots, it only showcases the construct. 
The malleability of identity and identification is made evident. 


Figure 21: Thomas Ruff, Venice Biennale, 1995. Images courtesy 
of the artist 


The scale of Ruff’s andere Portraits also runs counter to Galton’s be- 
liefs about the ideal way of perceiving composite portraits. Galton’s 
research mentions size as an important determinant of the read- 
ability of composite portraits, saying that, the smaller the image, 
the more likely it is that an observer will be able to see a single av- 
erage face emerge from the composite. Against this prescription, 
Ruff’s portraits instead actively make visible a constant movement, 
and seem to revel in the inability of the viewer confronting the por- 
trait to nail down anything “identifiable.” On this scale, the artifi- 
cial nature of the photographic image of a human subject gets exag- 
gerated, further emphasizing the limits of the photograph - that it 
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can only express the surface of things. Ruff’s large-scale exhibition 
of the andere Portraits reveals everything on the surface, making vis- 
ible the many layers and details of the images’ photographic struc- 
tures. And yet these images also evade any of the truth claims that 
are associated with the practice of mugshot portraiture from which 
he borrows. His work seems to state: just because something is seen, 
it does not mean it is true. In fact, his work goes further: the more 
you see, the less the truth can be revealed. 


Concluding Remarks 


Ruff’s andere Portraits are a perceptual study of the photographic 
construction of the face. The zones of incongruity between the fac- 
es are made visible, provoking a perceptual movement. This both 
reveals an internal relation between faces and produces a new, “fic- 
tional” face. Ruff recontextualizes the use of the portrait as a doc- 
ument, bringing the composite portrait into an art practice as an 
object of study and positioning it as a site of pure visuality. Yet it is 
still a portrait, and so the observer cannot help but connect the rep- 
resentation of the face with an identity. In this way, both the face 
and identity are revealed to be constructs. The binary presentation 
of faces provokes a constant perceptual movement - the face refus- 
es to stand still. The andere Portraits thus reveal a nuanced under- 
standing of the transience of identity, gender and form. In this way, 
Ruff’s artistic use of the composite is a distillation of the practice 
of representational resistance to reductive forms of recognition I 
discussed in the previous section. 

Through its appropriation of a security apparatus used for fa- 
cial recognition, Ruff’s artistic practice serves as a precursor to the 
artworks that I will go on to discuss. The andere Portraits experiment 
with the truth claims of the photographic document and visualize a 
dissonance between the subjective and objective processes of iden- 
tity construction through photographic representation. As an art- 
work, the portraits call our attention to the processes of perception 
both behind the image’s production and in the experience of the 
observer. Taken out of an original context of documentation, mug- 
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shots and security apparatuses, the andere Portraits point us toward 
the perceptual processes that are revealed upon the observation of 
the image, rather than depicting an image for a proposed function 
or to achieve a fixed outcome. Ruff’s portraits provide questions 
rather than answers. They question the dominance of the visual 
technologies used in the service of operations of recognition di- 
rected toward subjects. In particular, his composites constitute an 
artistic resistance to the visual act of classification. This theme of 
resistance to classification is taken up in the work of the following 
two artists, who in turn extend this line of questioning and critique 
towards the algorithmic forms of perception and recognition. 


Chapter 7: Faces in Excess: Zach Blas, 
Facial Weaponization Suite 


Background: A Conceptual Framework 


Discussing his artwork Facial Weaponization Suite (2014), Zach Blas 
states: “I saw a coterminous rise of masked protest alongside the 
rise and boom of biometric industries.” This connection between 
obscuring and scrutinizing the face in public life highlights the fact 
that the face is a site of confrontation between opposing political 
strategies. On the one side, institutional biometric technologies are 
employed as a means of regulating individuals through ascertain- 
ing their identities. On the other side, activist groups precisely resist 
identification and regulation as part of a political strategy. Through 
his art, Blas foregrounds a study of contemporary biometric facial 
recognition practices as rooted in the socio-political contexts in 
which they intervene. The series of masks produced as part of Fa- 
cial Weaponization Suite utilizes both a sculptural and a performative 
medium, through a series of workshops and events. Blas’ critique 
of biometric recognition involves understanding its technology 
as enacting a computational process of standardization and as an 
extended arm of neoliberal political strategies that unequally im- 
pact already vulnerable and marginalized groups in society.” Blas’s 


1 Ben Valentine, “Weaponizing Our Faces: An Interview with Zach Blas,” 
Vice, July 10, 2014, https://www.vice.com/en_us/article/vdpzaa/weaponi- 
zing-our-faces-an-interview-with-zach-blas-715. 

2 Talk given by Zach Blas as part of the symposium Invisible/Visible, held at 
the New Museum, New York, March 7, 2015. 
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artwork speaks directly to and confronts these aspects of biometric 
facial recognition, utilizing masks and performance in reference to 
a history of carnivalesque forms of hierarchical inversion. 

Blas’s critique of biometric technology is similar to that of schol- 
ar Shoshana Magnet, in particular to her call for a broader approach 
to technological failure.’ As Blas states: 


Biometric technologies rely heavily on stable and normative con- 
ceptions of identity, and thus, structural failures are encoded in 
biometrics that discriminate against race, class, gender, sex, and di- 
sability. [..] Biometric failure exposes the inequalities that emerge 
when normative categories are forced upon populations.* 


Blas’s work is informed by the theoretical frameworks of queer the- 
ory and post-colonial theory, utilized as critical tools through which 
to approach technology. What Blas argues is that biometric recog- 
nition involves a computational process of standardization with set 
parameters produced through the biases inherent in a history of 
social discrimination. This computational process, Blas argues, has 
deeply embedded within it a visuality drawn from societal norms of 
gender, race and sexuality. 

Blas’s work speaks to a shift in political strategy in relation to 
activism and civil disobedience. In the 1970s in the US, the tools 
of visibility and political representation were the primary vehicles 
of political agency for people of color, gay activists and feminist 
groups. Visibility and political representation were, in this context, 
equated with political agency and legitimization, providing indi- 
viduals from marginalized communities with a platform to com- 
municate and fight for specific political goals. Along with certain 
other artists, Blas argues that, with the increasing pervasiveness 
of contemporary surveillance technologies such as biometrics, visi- 
bility and representation have instead become a tools of regulation 


3 Magnet, When Biometrics Fail, 9. 
4 Zach Blas, Facial Weaponization Communiqué: Fag Face (2012), video, https:// 
vimeo.com/57882032. 
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and control wielded by the state.* These technologies enact a form of 
societal control through the increasing accumulation of data pro- 
duced from surveilled bodies. Blas states: 


Such a digital regime profoundly inverts the political promise of vi- 
sibility and representation as means toward democracy and equa- 
lity. Any exposure of bodies is now usurped as a potential pathway 
to control and governance, and thus, undoes documentation as a 
purely liberatory project.® 


The contemporary techno-political landscape of pervasive sur- 
veillance technologies and data accumulation inverts the political 
agency of visibility. This inversion, in turn, produces an alternative 
form of contemporary political agency that is rooted in the claim to 
the right not to be visible, the right not to be recognizable or made 
vulnerable to exposure — to be able to protect one’s identity from 
automated enrollment in biometric recognition operations. The 
threat of discrimination that unequally affects marginalized com- 
munities persists and, in many ways, has not changed through a 
long history of racial and gendered discrimination in the US, but 
the technologies of control and regulation have. Biometric recogni- 
tion enacts a negation of certain identities according to normative 
categories, prohibiting individuals from certain spaces and actively 
discriminating against certain groups. This change demands new 
and revised strategies of political agency. 

Blas draws his central concept of “opacity” from the work of the 
Martiniquan philosopher and poet Edouard Glissant, who was ac- 
tive in the anti-colonial movements of the 1950s and 1960s. The art 
historian and novelist Teju Cole, in a New York Times article on pho- 
tography and the representation of black skin, offers this insightful 
description of the term: 


5 See also artist, Paolo Cirio, Obscurity (2016), inkjet prints, https://paoloci 
rio.net/work/obscurity/. 

6 Zach Blas, “A Cage of Information, or, What is a Biometric Diagram?” in Do- 
cumentary across Disciplines, ed. Erika Balsom and Hila Peleg (Cambridge, 
MA: MIT Press, 2016), 82. [80-90] 
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Glissant defined it as a right to not have to be understood on others’ 
terms, a right to be misunderstood if need be. The argument was 
rooted in linguistic considerations: It was a stance against certain 
expectations of transparency embedded in the French language. 
Glissant sought to defend the opacity, obscurity and inscrutabili- 
ty of Caribbean blacks and other marginalized peoples. External 
pressures insisted on everything being illuminated, simplified and 
explained. Glissant’s response: No.” 


Blas situates Glissant’s concept of opacity within a contemporary, 
Information Age techno-political landscape and applies it to the 
language of data production - specifically to the surveilling of 
bodies and the transformation of physical embodiment into data. 
In the contemporary context, the “expectations of transparency” 
and an insistence on “everything being illuminated, simplified and 
explained” is enacted through the “forced visibility” of individu- 
als through surveillance and biometric recognition technologies. 
Marginalized groups, such as people of color, immigrants and the 
LGBTQ community, are particularly vulnerable to state regulatory 
practices and standardization practices, which negate their identi- 
ties through biometric recognition while, at the same time, forcing 
them to be, as Blas puts it, “informatically visible.” Through the use 
of Glissant’s concept of opacity, the right not to be seen takes on an 
ethical and concrete significance. In a nod to Glissant, Blas coins 
the term “informatic opacity,” which he defines as 


a refusal to visually cohere to digital surveillance and capture tech- 
nologies’ gaze. A theory and practice whose goal is maintaining the 
autonomous determination of alterity and difference [..] evading 


7 Teju Cole, “A True Picture of Black Skin,” The New York Times Magazine, Feb- 
ruary 18, 2015, https://www.nytimes.com/2015/02/22/magazine/a-true-pic 
ture-of-black-skin.html. 

8 Zachary Marshall Blas, “Informatic Opacity: Biometric Facial Recognition 
and the Aesthetics and Politics of Defacement,” (PhD diss., Duke University, 
2014), iv. 
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the quantification, standardization, and regulation of identity im- 
posed by biometrics and the state.’ 


In this way, informatic opacity can function as both a tactic anda 
material condition. It can serve as a tool of political activism, with- 
in the discourses of identity politics, in enabling people to oppose 
a politics of identification. The right not to be visible equals a right 
to self-determination, to be able to “self-generate” an identity that 
would otherwise be negated through processes of biometric recog- 
nition. 


Facial Weaponization Suite (2011-2014) 


Blas produces a materialization of “informatic opacity” in his 
masks and through a vehicle of “excess.” Blas describes how he 
constructs the masks from the faces of multiple participants who 
attend workshops he runs in collective art spaces. The participants 
at the workshop agree to be scanned using a Kinect 3D scanning 
device, which yields data relating to each person’s face. Blas then 
collaborates with a modeler to generate a mold of a mask from 
the compiled facial data from all the participants. Blas states that 
this facial data is “not averaged” but rather compiled into a form- 
less shape. He describes how this results in an “amorphous mask 
that resembles only abstract surfaces,” making it “biometrically 
unrecognizable.”° The mold is produced through a process of CNC 
milling, that is, “computer numerical controlled” machining, which 
relies on programmed code to determine the movements that cre- 
ate the mold. This method allows for a high degree of precision in 
movement. CNC milling utilizes a rotating cylindrical cutting tool; 
the piece is moved across the milling tool in different directions, 
making it possible to create the amorphous, customized shapes 
of Blas’s masks. This mold is then used to vacuum form multiple 
masks. 


9 Ibid. 
10 Blas, Informatic Opacity, 78. 
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Blas has exhibited both the masks themselves and highly styl- 
ized studio color photographs of the masks. The masks themselves 
are glossy, highly reflective and brilliant in color. The monochrome 
colors of the masks are significant. Most of Blas’s masks are simply 
titled Mask, with the date and location of the mask’s production, but 
the colors allude to specific cultural contexts and symbolize the nor- 
mative categories that these masks work to obfuscate. For example, 
the creation of a blue mask came out of a workshop Blas held that 
was attended by artists, intellectuals, curators, activists and tech- 
nologists on the subject of feminist theory. The workshop covered 
issues of visibility, recognition and concealment, and a particular 
issue arose concerning the use of the veil by Muslim women and 
how this “complicates western feminism’s investment in visibility 
politics." They discussed a particular incident that had become a 
kind of inflection point in relation to these issues. The incident oc- 
curred in 2011 during the uprisings in Cairo, Egypt, and is referred 
to as the “blue bra” incident. A woman wearing an abaya was partic- 
ipating in the demonstrations in Tahrir Square, and she was brutal- 
ly beaten by Egyptian soldiers. As she was dragged off, her clothes 
were ripped away, revealing a blue bra (figure 22). The image of the 
blue bra became a feminist rallying symbol for Egyptian women 
protesting oppression.” In reference to this incident, the color blue 
was chosen for the mask that was produced following the workshop 
(figure 23). The form and shape of the mask also recalls the veil worn 
by Muslim women. The use of blue in Blas’s mask symbolizes the is- 
sues that sit at the intersections of feminist resistance in the cultur- 
al context of the Middle East, the wearing of the veil and investment 
in both concealment and exposure. 


11 Ibid., 163. 

12 Kainaz Amaria, “The ‘Girl in the Blue Bra,” NPR, December 21, 2011, 
https://www.npr.org/sections/pictureshow/2011/12/21/144098384/the- 
girl-in-the-blue-bra. 

13 Blas, “Informatic Opacity,” 162. 
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Figure 22: Egyptian army soldiers arrest a female protester during 
clashes at Tahrir Square in Cairo on Dec. 17. © Stringer/Reuters/Landov 


Figure 23: Zach Blas, Mask- November 20, 2013, New York, NY, 
Facial Weaponization Suite, Photo by Christopher O'Leary. 
Image courtesy of the artist 
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Figure 24: Zach Blas, Fag Face, Facial Weaponization Communiqué: 
Fag Face, video still (2012) Image courtesy of the artist 


d 


The first mask Blas created in the suite is one of the few to be given 
a name other than “mask.” It is titled Fag Face (2012) (figure 24). (The 
figure shows an image of a virtual model of the mask, taken as a 


video still from Blas’s short film “Facial Weaponization Communi- 
qué: Fag Face.”) In constructing this mask, Blas had in mind certain 
scientific studies, such as one conducted at Tufts University, on the 
recognition of homosexuality. These studies tested for an ability to 
identify homosexuality through a rapid recognition exercise using 
photographs of men’s faces. There has since been much controversy 
over similar research into the machinic recognition of homosex- 
uality, in particular studies conducted at Stanford University by 
Michal Kosinski and Yilun Wang." Blas zeros in on the most per- 
tinent question one might have upon learning of the existence of 
these studies, which is: why? To what ultimate end will these tests 
contribute? Of all the possible purposes to which these tests could 
contribute, the primary and most obvious is the control and regu- 
lation of homosexuals through the utilization of the technological 
acts of sorting and categorizing. Blas constructed the mask Fag Face 
as a direct response to the scientific studies at Tufts on rapid facial 
recognition of sexual orientation. Like the other masks in the suite, 
it was generated by scanning multiple faces, but in this case specif- 


14 See “Keeping a straight face,” The Economist, September 9, 2017, 67-68. 
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ically gay men’s faces. The resulting physical mask is a candy-pink 
blob of gloss (figure 25). Although constructed from data about the 
faces of gay men, this mask obscures the identity of each individual 
face and that of the wearer of the mask. In response to the Tufts test, 
the mask accelerates the “rapid recognition” aspect by expressing 
the faces of multiple gay men at once. Through both the elucidative 
material of the mask and its instantaneous projection of a multi- 
tude of facial forms, it renders a kind of play between full-on visi- 
bility and concealment. 


Figure 25: Zach Blas, Fag Face Mask — October 20, 2012, Los Angeles, 
CA. Facial Weaponization Suite. Photo by Christopher O’Leary. Image 
courtesy of the artist 
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On another level, Blas’s work is performative. After a mask is com- 
pleted, the workshop participants create a masked public perfor- 
mance that highlights the inequalities inherent in biometric facial 
recognition use. Blas has chosen to hold these performances in ar- 
eas of particular relevance to the issue of facial recognition. For ex- 
ample, one performance, titled Procession of Biometric Sorrows, was 
held at the US—Mexico border on June 5, 2014. (figure 26). It called 
attention to the immense amount of biometric data that is gathered 
at borders and in particular at the US—Mexico border. Blas notes 
that a central subject of discussion in the workshop preceding this 
performance was the fact that the Mexican government had recent- 
ly introduced biometric identification cards for children.” 


Figure 26: Zach Blas, Facial Weaponization Suite: Procession of 
Biometric Sorrows, Museo Universitario Arte Contemporáneo (MUAC) 
Mexico City, Mexico (5 June 2014). Photo by Orestes Montero Cruz. 
Image courtesy of the artist 


15 This issue of biometric identification cards for children foreshadows the 
more recent incidents of mistreatment of Mexican children at the US-Me- 
xico border. Children have been separated from their parents and detai- 
ned by US authorities in camps at the border, a practice that has been un- 
derstood as a gross abuse of power by the Trump administration. 
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A Collective Excess 


These performances, as wellas the creation of the masks themselves, 
highlight Blas’s collectivist strategy. In drawing on this strategy, he 
references contemporary political movements that rely on a social 
collectivity to confront the widespread abuse of institutional and 
governance powers, such as the Occupy movement, Pussy Riot and 
the Zapatista movement in Mexico. These particular movements 
rely on a strategy of masking or facial concealment: Occupy with 
the use of the Guy Fawkes mask, the pink baklavas of Pussy Riot and 
the Zapatistas’ bandanas, which conceal the lower half of the face. 
Blas’s reference to a “weaponization” of the face is an acknowledg- 
ment of these movements; what he calls the “power of the collective 
face” merges a socio-political apparatus of collectivity with faciality. 
He is pinpointing the face as a site of politics. In these movements, 
the face is recreated; it is reconstructable and interchangeable, 
rather than being a source of recognition and thereby a means of 
regulation by the state. Removing the recognizable features of the 
face, the members of these movements become a faceless threat to 
the asymmetrical systems of power that they confront. In this con- 
text, facelessness is a source of power. The use of masks in Blas’s 
performances draws on a historical use in the context of the car- 
nival, where members of the populace were able to speak truth to 
power and where social and political hierarchies were inverted 
through satire. In this context, masked performances play with 
the distinction between what is seen and unseen. They at once make 
the wearer highly visible, giving their performer a platform, and at 
the same time erase individual identity. This allows for the wearer 
of the mask, and so the speaker of truth, to be anonymous. 

Blas’s masks are constructed from the forms of many faces, and 
as such they merge the individual wearer of a mask with the masses. 
In this way, for me, the masks recall Philip K. Dick’s “scramble suit” 
in his psychotropic sci-fi novel A Scanner Darkly. In Dick’s novel, the 


16 The use of masks in these sorts of contexts spans many different cultu- 
res. See, for example, the use of “tal” masks in Korea; the etymology of the 
word derives from the Chinese character meaning “to be free.” 
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“scramble suit” is worn by narcotics officers, and thus is a technolo- 
gy of policing. He writes: 


The scramble suit was an invention of the Bell laboratories, conju- 
red up by accident by an employee named S. A. Powers [...] Basically, 
his design consisted of a multifaceted quartz lens hooked up to a 
million and a half physiognomic fraction-representations of vari- 
ous people: men and women, children, with every variant encoded 
and then projected outward in all directions equally onto a super 
thin shroudlike membrane large enough to fit around an average 
human. 


As the computer looped through its banks, it projected every con- 
ceivable eye color, hair color, shape and type of nose, formation of 
teeth, configuration of facial bone — the entire shroudlike membra- 
ne took on whatever physical characteristics were projected at any 
nanosecond, then switched to the next [...] 


In any case, the wearer of ascramble suit was Everyman and in eve- 
ry combination (up to combinations of a million and a half sub-bits) 
during the course of each hour. Hence, any description of him — or 
her—was meaningless.” 


The description of the “scramble suit,” with the “everyman” pro- 
jection of every face obscuring the wearer’s identity, could be a 
description of Blas’s masks. Blas’s masks, like the “scramble suit,” 
present a representation of multiple identities as a strategy of cam- 
ouflage. It is interesting to think about the use of “Everyman” in 
Dick’s scramble suit in relation to Blas’s strategy of “collectivism.” 
Both erase any characteristic features through an excess of features. 
In other words, in showing too much one cannot see anything at 
all. This is a tool used by certain artists in the Information Age’ to 


confound systems of surveillance and control. It is a tool that makes 


17 Philip K. Dick, A Scanner Darkly (New York: Vintage Books, 1991), 22-23. 


18 See work of artist Hasan Elahi. Hasan Elahi website, accessed June 3, 2019, 
http://elahi.umd.edu and Karen Kedmey, “Hasan Elahi vs. The FBI: The Art 
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use of the original material of surveillance, that is, the information 
itself, and overloads the system with it. Like Dick’s “scramble suit,” 
which “projected every conceivable eye color, hair color, shape and 
type of nose, formation of teeth,” Blas’s masks use the distinctive 
characteristics and endless variability of the face as information, 
before combining it in order to produce indistinction. Blas’s masks 
project this mutability as an opaque form all at once, however, rath- 
er than in nanoseconds. Instead of a multifaceted quartz lens, the 
material of Blas’s masks is opaque and reflects only light, rather 
than faces, from off of its glossy surface. This difference also re- 
veals that Blas’s masks play with the notion of the surface. What 
can be seen in his masks is only surface, and because that surface is 
highly glossed and vividly colored, the observer’s attention is drawn 
to it rather than to anything underneath. This mirrors the fact that 
facial recognition technologies can only recognize surface qualities 
and that surface can be dressed up however an individual chooses. 
In Dick’s novel, this is used as a method of concealing the identi- 
ties of members of the police. In Blas’s work, it is used as an aes- 
thetico-political tool to conceal the identity of the individual from 
dominant systems of population control. Instead of being a tool of 
policing, Blas’s masks are a weapon for the proletariat. As the title 
of Blas’s project makes clear, when used as a tool of resistance, the 
collective is the weapon. 

In their opposition to standardization and in their formless- 
ness, Blas’s masks exhibit a strategy of excess. Blas has described 
his masks as a “collective excess, that exceeds the boundaries of the 
individual.” In a written piece titled “Fag Face,” Blas has explained 
this excess further, using terms that are physical, subjective, bodily 
and embodied. Excess is utilized here as a defiant response to the 
inherent violence in the use of the term “fag” as a derogatory label 
for gay men: 


of Self-Surveillance,” Artsy, May 27, 2016, https://www.artsy.net/article/ 
artsy-hasan-elahi-vs-the-fbi-the-art-of-self-surveillance. 


19 Blas, “Informatic Opacity,” 79. 
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Fag face captures me into an identity that is not my own, a grid that 
legislates me. 


How do I escape this face? How do I desire to escape this gridding 
of my head? How can I open, make into a mystery, liberate my fag 
face? 


Force the face forward. Bring the face to the limit of these grids, so 
that it can de-code its boundaries, break them open, to enter again 
into the swarming chaos of matter that resists recognition. 


If fag fucking is what fag face visualizes to the other, push this 
further. 


Accumulate cum so that your face becomes a volatile liquid surface 
with no eyes, nose, or mouth; keep the smell from rimming so that 
your face and ass are irreducible; let the pubic hair gather into dif- 
ferent consistencies of stickiness; wipe the shit left on your fingers 
under your hidden, cum-filled eyes like war paint. Transform your 
face into a hypertrophized state of fag-ness. Let these new excesses 
dissolve readability. Let your fag face configure with these mate- 
rials into that which is not identifiable 


Once 1000 cocks have cum on my head and 1000 asses have wiped 
their shit and sweat there, try to tell me what my face is.”° 


Like his masks, Blas’s text opposes the standardization mecha- 
nisms of facial recognition through a tactic and materialization of 
opacity. In this case, opacity comes in the form of excess, an ex- 
cess expression of that which underlies homophobia. Opposing the 
mechanized and automated processes of a biometric operation, this 
text, as lyrical poem, is unapologetically human in all its corporeal 
glory, both in flesh and liquid form, and in its tenor of punk-rock 
revolt. The text references the bodily sense that cannot be measured, 


20 Zach Blas, “Fag Face,” Recaps Magazine, accessed May 3, 2019, http://re 


capsmagazine.com/review/fag-face-by-zach-blas/. 
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including desire. The text is not only a statement of resistance to an 
operation of facial recognition but confronts the individuation of 
recognition and the regulation of gay male sexuality with an excess 
of it: I will meet your homophobia with my cock — 1000, to be pre- 
cise. Strength in numbers. Through Blas’s textual work and in his 
production of masks, he inverts an essential dilemma and demand 
to which biometric technology is so often called upon to respond — 
the problem of volume, or specifically the problem of ascertaining 
identity from a massive amount of information. Blas utilizes the 
problem of volume and transforms it into an aesthetic, solidifies it 
in sculptural form as a materialization of opacity. 

Blas’s artwork is particularly significant when considered in re- 
lation to the eigenface algorithm and the composite form analyzed 
in the previous section. Through his work, the composite form itself 
is made operational. His approach to designing the masks referenc- 
es ahistory of facial recognition techniques, as he makes clear in his 
dissertation.” Blas directly references the work of Francis Galton 
and his composite portraits. He describes his masks and the cor- 
responding processes of “digital collectivization” as the “antithesis” 
to Galton’s composites of criminals and production of types.” He 
states: “In contrast, the collectivizing process in Facial Weaponiza- 
tion Suite reveals that Galton’s compositing method can be employed 
to arrive at the exact opposite of his intentions [...] it can also gen- 
erate a collective excess that exceeds the boundaries of the indi- 
vidual.” This brings to mind a passage from Galton’s description 
of his composite portraiture practice: “No statistician dreams of 
combining objects into the same generic group that do not cluster 
towards a common centre; no more should we attempt to compose 
generic portraits out of heterogeneous elements, for if we do so the 
result is monstrous and meaningless.”* With his sculptural com- 
posites, Blas injects meaning into the “monstrous.” Although Blas 
sees his work as the antithesis of Galton’s production of a type, his 


21 Blas, “Informatic Opacity.” 
22 Ibid., 79. 
23 Ibid. 


24 Galton, Inquiries, 230. 
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practice nonetheless involves, I argue, a similar process of produc- 
tion and creates a similar structure to Galton’s composite. In the 
bringing together of people under categories of identity such as 
“gay men,” Blas also produces a composite. It is this similarity of 
structure that makes Blas’s work a fascinating study in this regard: 
Blas then takes the composite form and produces something that is 
the opposite of a “type.” Blas’s rejection of Galton’s type through his 
masks and through what he calls the “collective excess that exceeds 
the boundaries of the individual” results in a sculptural formation 
of the Galton composite. 

The masks visualize something very similar to Wittgenstein’s 
concept of “aspect perception.” Both “aspect perception” and “col- 
lective excess” turn the aesthetic focus of the composite form on to 
the mutability and fluidity of forms. They both reveal a kind of re- 
lationship between disparate forms. The notion of collective excess 
operationalizes the notion of aspect perception in contemporary 
politics. Blas’s masks can be understood, in this way, as solidify- 
ing a mutability of forms and activating the composite precisely to 
resist an automated (and reductive) recognition process. To return 
to the topic of eigenface, we might imagine Blas’s masks as what 
would happen were one to produce an eigenvector in physical form 
and then reflect it back into the eigenface algorithm. In this way, I 
see Blas as taking the output data from an algorithm and projecting 
it on to its front end. It is like an algorithmic mirror in which the 
algorithm cannot recognize itself or what it has produced. It is as if 
the algorithm cannot read its own form of representation. 


Concluding Remarks 


Blas’s Facial Weaponization Suite is not practical; it does not offer a 
strategy for actual informatic opacity in the face of non-consensual 
biometric enrollment — unless, that is, we were all to walk around 
with masks over our faces in our daily lives. Yet what I find most im- 
portant about his piece and the many forms it has taken - workshop, 
performance, short film, text and masks - is that they all point to- 
ward the production of another form of subjectivity, one that is the 
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result of a “collective excess.” Discussing Facial Weaponization Suite, 
Blas has said that the masks “articulate a presence.” The pres- 
ence is formed through a collective. In Blas’s strategy of negating 
the process of biometric recognition — albeit in a symbolic fashion, 
through performativity and masking - there is the formation of a 
kind of collective subjectivity, one that is not defined by individu- 
ation. This has far-reaching implications for our understanding 
of the parameters of future forms of political, ethical and social 
organization. Blas’s masks are a conglomerate of a multiplicity of 
angles and perspectives from different faces, depicting a kind of 
endless variation. Their construction draws on the concepts, strat- 
egies and discourses of queer theory, feminist critical theory and 
post-colonial theory. They propose an alternative representational 
mechanism to the one found in AFR systems, a mechanism that 
captures a mutable human form. The result is a representation of a 
subjectivity that is manifold and self-generated. The representation 
found in Blas’s work is of a collective figure, reflecting a collective 
subjectivity. 


25 Valentine, “Weaponizing Our Faces.” 


Chapter 8: An Algorithmic Ready-made: 
Trevor Paglen, Adversarially Evolved 
Hallucination and Eigenface 

(Even The Dead Are Not Safe) 


Background: Notions of Transparency 


While Blas’s work references a notion of opacity, the work of the art- 
ist Trevor Paglen conjures up its opposite, transparency. Much of 
Paglen’s work involves the production of images in environments 
and milieus where there has been a conscious effort to suppress 
images. Paglen’s artwork has exposed the nature of opacity in co- 
vert military and political agencies, such as the NSA and the CIA, 
revealing a surrealist character to their agendas and the physical 
structures through which they operate, forcing a kind of trans- 
parency into their workings. His recent works concern the topic 
of machine vision and its increasing prevalence in society. He ap- 
proaches the topic of machine vision as he does his other subject 
matter, exposing the hidden and obscure processes through which 
it operates and creating spaces of transparency and visuality. What 
particularly interests Paglen about machine vision is the growing 
economy of images, produced by and for machines, that is evolving 
and, for the most part, going unseen by the human audience. His 
inquiry acknowledges the influence of machine vision in bringing 
about a transformation in contemporary visual culture as a whole, 
where increasingly, he argues, human vision is an exception to the 
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rule.' And yet, as Paglen argues, these machine vision images play 
an increasingly large role in human interactions, with “their func- 
tions changing from representation and mediation, to activations, 
operations, and enforcement,” thus making it necessary for us to 
scrutinize not only the images but the processes through which 
they enact a form of vision.” 

In his attempts to reveal the inner workings of forms of machine 
vision, Paglen produced a series of images using different artificial 
intelligence (AI) systems of machine vision. These works culminat- 
ed in an exhibition at Metro Pictures Gallery called A Study of Invis- 
ible Images.* Some of the works included in this exhibition appro- 
priate the actual processes of algorithms that have been designed 
to enact different operations of machine vision. Two such works 
will be discussed here: a series titled Adversarially Evolved Hallucina- 
tion (2017) and a series of works made with the eigenface algorithm 
titled Eigenface (Even The Dead Are Not Safe) (2017).* In both of these 
works, Paglen approaches the technology of image recognition sys- 
tems (with the latter dealing directly with a facial recognition sys- 
tem) and provides a kind of visual dissection of their algorithmic 
processes. 

In producing these works, Paglen is part of a growing movement 
of artists using code to make art. I would describe Paglen’s artistic 
approach to image and face recognition technology as internal; that 
is, he uses the algorithmic processes themselves as material for the 


1 Trevor Paglen, “Invisible Images (Your Pictures Are looking At You),” The 
New Inquiry, December 8, 2016, https://thenewinquiry.com/invisible-ima- 
ges-your-pictures-are-looking-at-you/. 

2 Ibid. 

3 Trevor Paglen, A Study of Invisible Images, exhibition, September 8-October 
21, 2017, Metro Pictures Gallery, New York, NY. 

4 Paglen included other works in this exhibition that refer directly to diffe- 
rent facial recognition systems, including It Began as a Military Exhibit and 
Machine-Readable Hito. 1 chose the two works discussed here because of 
their connection and relevance to the overall analysis. Another of Paglen’s 
works that bears on facial recognition (specifically automated facial-beha- 
vioral analysis) is his piece Sight Machine, performance, January 14, 2017, 
San Francisco, CA. 
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production of images. In this way, his artwork is less a confronta- 
tion with the technology's front-end operations and more an en- 
gagement with the internal workings of how the technology enacts 
recognition. This approach in turn, alters the production and oper- 
ation of the algorithm itself. This contrasts with Blas’s work, where 
our attention is largely drawn toward a confrontation between the 
recognition device and the embodied subject under surveillance. In 
presenting images that express how the internal workings of AI sys- 
tems see, Paglen’s work has a way of being pedagogical, teaching an 
observer of his images about how an algorithm sees. In the absence 
of further explanation, this aspect can make his work somewhat 
opaque and hard to penetrate. 

One of the strengths of Paglen’s work is its ability to put tech- 
nological processes into dialogue with discourses in art history and 
political philosophy. In doing so, his works not only expose but also 
actively reimagine the technological processes at issue and hold out 
the promise of a reinterpretation of the possibilities of the tech- 
nology’s operations. Paglen finds unintentional affinities between 
abstract art and linear classifiers — the archetypes used to train al- 
gorithms how to recognize objects. I see Paglen’s use of algorithmic 
processes to produce images within a cultural context, in fact, as a 
reference to another art-historical practice, that of the ready-made, 
a method of art production involving found but modified objects/ 
products, first used by Marcel Duchamp, who presented manufac- 
tured objects as objects of art. An aspect of the ready-made is the 
use of ubiquitous, everyday objects. In the contemporary world, al- 
gorithms have become one such everyday object; although they are 
immaterial and their presence undetectable, they are ubiquitous at 
all levels and areas of interaction. Paglen’s appropriation of image 
recognition algorithms treats them as an immaterial ready-made: 
that is, he uses preprogrammed or “found” algorithms and modifies 
their output, positioning this output within a discourse of art pro- 
duction and within various conceptual frameworks. In producing 
an “algorithmic ready-made,” Paglen takes the everyday object of 
the algorithm and merges its technical processes with the concep- 
tual. 
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Adversarially Evolved Hallucination (2017) 


Paglen produced the series of works titled Adversarially Evolved Hal- 
lucination during a residency at Stanford University. In referring to 
these images as “hallucinations” generated from machine-vision 
processes, Paglen characterizes them as departing from reality — as 
though we are about to see what machines see while on acid. There 
is, he is saying, an illusory quality to the machine-vision process- 
es usually deemed objective, scientific and engaged in definitive 
operations. The word “adversarial” in the title of the series refers 
specifically to images that are developed by computer scientists to 
trick image recognition systems. These “adversarial images” are de- 
veloped to exploit weaknesses in the recognition algorithm, mak- 
ing them see things that are not there. The purpose of these images 
is to better train the algorithm - a kind of exercise of productive 
technical failure. These adversarial representations eventually get 
incorporated into the training sets used by the algorithm in ma- 
chine learning. With the help of a small team of developers, Paglen 
customized the software used for recognition and developed soft- 
ware suites. He used two separate algorithmic processes: one that 
taught the machine to recognize objects through training sets and 
the other that was used to generate adversarial images based on the 
same training set. Paglen referred to the secondary algorithm as 
the “Generator,” while the first he called the “Discriminator.” Pa- 
glen designed these two algorithms to engage in a dialogue with 
each other: to play a game in which the second algorithm attempted 
to fool the first through the production of adversarial images. Pa- 
glen explains: 


The two Als go back and forth thousands or millions of times, until 
the Generator has learned how to make images that can reliably 
fool the Discriminator. The images that come out of this process are 
called Hallucinations. Together, the Als have evolved an image that 


5 Metro Pictures, Trevor Paglen: A Study of Invisible Images, (New York: Metro 
Pictures, 2017), 24. 


6 Ibid. 
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is entirely synthetic and has no referent in reality, but that the pair 
of Als believe are examples of things they’ve been trained to see.’ 


The resulting images are surreal, pixelated forms that emerge in 
vivid color. The game that the two Als play is not only a way of ex- 
ploiting the weaknesses and parameters of algorithmic learning; it 
is also a game that plays with the idea of recognition itself. How 
does the AI know something as it is? What can be recognizable? 
In this algorithmic dialogue, Paglen toys with the relationship be- 
tween visual form and knowledge. A key aspect of these works is 
found in their titles, which are taken from the categories of training 
set images used by the algorithms. Instead of the phrase “training 
set,” Paglen uses the term “corpus,” bringing attention to the role 
of training sets as a body of knowledge. The corpuses that are re- 
ferred to in the titles include “Omens and Portents,” “Monsters of 
Capitalism,” “American Predators” and “The Aftermath of the First 
Smart War.” These categories organize the types of objects being 
recognized within Paglen’s software suite, and they clearly differ 
from the actual categories of training sets used in machine image 
recognition. Instead, they reflect concepts that act as metaphors, 
describing the wider socio-political contexts in which the technol- 
ogy is used. The images that result from this algorithmic game of 
recognition render these metaphors visible. For example, one im- 
age, Highway of Death (Corpus: The Aftermath of the First Smart War) 
(figure 27), references the militaristic context of the use of image 
recognition. The image is a surrealistic rendering of a desert bat- 
tlefield with no humans present. What is present in the image is a 
shift in replacing the categories of training sets to recognize objects 
or people to the recognition of the cultural contexts of its operation. 
Through Paglen’s production of adversarial images, he is experi- 
menting with training an algorithm (and the human observer of the 
generated image) to not only see objects but also to see concepts that 
shed light on the wider cultural contexts in which these technolo- 
gies intervene. 


7 Ibid. 
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Figure 27: Trevor Paglen, “Highway of Death (Corpus: The Aftermath 
of the First Smart War)” Adversarially Evolved Hallucination, 2017. 
dye sublimation metal print. Image courtesy of the artist 


For Paglen, the importance of developing a vocabulary through 


metaphor connects back, I think, to a previous series of his, Symbol- 
ogy (2006), in which he collected material paraphernalia relating to 
black ops and covert activities of the CIA, including fabric patches, 
designed by members of covert units, that are embroidered with 
coded insignia, symbols such as dragons, arrows, animals, planets 
and phrases such as “We Own the Night.” The character of the meta- 
phoric forms and titles found in the images of Adversarial Hallucina- 
tions may be seen as mirroring these ominous symbols and phrases. 
Both reflect a world that would otherwise not be seen, and once 
exhibited become objects of conceptual inquiry. Paglen describes 
the patches used in Symbology as follows: “A part of the military's 
everyday culture [...] If you could begin to learn its grammar, you 
could get a glimpse into the secret world itself.” This description 


8 Trevor Paglen, I Could Tell You But Then You Would Have to Be Destroyed By Me: 
Emblems From the Pentagon's Black World (Brooklyn: Melville House, 2010), 4, 5. 


Chapter 8: Trevor Paglen, Adversarially Evolved Hallucination and Eigenface 


can be extended, I think, to his approach in Adversarial Hallucina- 
tions, which develops a visual vocabulary of algorithmic processes, 
building on a taxonomy of corpuses, providing a grammar to see 
how machines see within the contexts in which they operate. 

Some of the titles of these corpuses refer to literary metaphors. 
Through the corpus titles, Paglen refers to a conceptual taxonomy 
based in literature and philosophy. As I have mentioned, one of 
the corpuses references the allegorical monsters of capital,’ that 
is, monsters that have acted as literary metaphors for capitalism. 
One in particular, takes the form of a face - Paglen’s piece Vam- 
pire (Corpus: Monsters of Capitalism) (figure 28). The image is pretty 
terrifying. It is of a face that appears to be constructed from melt- 
ing pixels. It contains all the facial landmarks: eyes, nose, cheeks 
and mouth. Yet these landmarks are composited together from 
disparate sources and fragments of light and shadow, creating a 
kind of digital collage work. The left eye resembles a photograph of 
an actual eye seemingly cut out from a magazine, and it peers out, 
furtively, from a dark curtain of velvet blur. From this eye, bluish 
veins appear to drip down. The other eye, in contrast, is constituted 
by a dark swirl resembling a wilting red carnation. A jagged light- 
ning bolt parts light and shadow across the face and acts to form 
the nose. In place of a mouth is a kind of striated, conglomerate of 
blood-red, pixelated ooze. Much of the face is hidden in shadowless 
depths of darkness. In the areas oflight, specifically in the forehead, 
one can detect a painterly effect of the multiple layers of digital im- 
agery that constitute this composited face. The predominance of a 
Dada-like palette of black, white and red contributes to the stark na- 
ture of this vampire face. The more realistic left eye is the only part 
of the face that is a recognizable, complete form. Without that eye 
one could hardly see a face in this image. That eye is thus both an 
organizing feature and also the source of the image’s terror. For, in 
its fully realized form, it is what allows the vampire/monster in the 
image to look back at the observer, and as an observer we feel very 
much under its mono-gaze. 


9 For more on this subject see David McNally, Monsters of the Market: Zombies, 
Vampires, Global Capitalism (Chicago, IL: Haymarket Books, 2012). 
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Figure 28: Trevor Paglen, “Vampire (Corpus: Monsters of Capitalism)” 
Adversarially Evolved Hallucination, 2017. dye sublimation metal print. 
Image courtesy of the artist 


Paglen has described how he thinks of AI itself as a monster of 
capitalism. The subject of the image in Vampire (Corpus, Monsters of 
Capitalism) alludes to Karl Marx’s vampire of dead labor: “Capital is 
dead labour, that, vampire-like, only lives by sucking living labour, 
and lives the more, the more labour it sucks.” As such, the image 
references a wider context in which market forces drive image-rec- 
ognition algorithms, and AI in general, to be deployed in order to 
replace human labor. In his allusion to Marx and with his depiction 
of a vampire as one particular monster of capitalism, his work may 
be seen as a critique of the forces and agendas behind the technolo- 
gy’s development and use — politically, commercially and militarily. 


10 Karl Marx, Capital: A Critique of Political Economy, vol. 1, trans. Samuel 
Moore and Edward Aveling, (Moscow: Progress Publishers, 1887), 163. 
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His piece visualizes the “face” of this “monster of capitalism,” and as 
such it stands out in the context of this thesis on facial recognition. 
For this is another construction of a portrait by an algorithm, yet it 
is one based on and born entirely from a concept. In this way, Pa- 
glen also says something about faces as conceptual constructs. This 
face was constructed, as were the other images in the series, from 
a training set of thousands, if not millions, of collected images — in 
this case, various images of vampires. 


Figure 29: Trevor Paglen, training set images for “Vampire (Corpus: 
Monsters of Capitalism)” Adversarially Evolved Hallucination, 2017. 
Image courtesy of the artist 


© 
2E HA” tnra 
DOE : Ree 


Figure 29 shows a sample of the training library, revealing a wide 
variety of cultural imaginations of vampires, ranging from the be- 
loved Count von Count Muppet from Sesame Street, to photographs 
of infant vampires, to depictions of vampiric motifs in fashion 
spreads and medieval paintings. 
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In focusing on the initial training-set phase of the recognition 
process, Paglen problematizes a central issue with automated rec- 
ognition through AI: the way algorithms learn through training sets 
and, as such, fix meaning in the physical world. Paglen describes 
the training-set process as follows: 


[t]his all takes place largely for the most part imperceptibly. Assig- 
nations of meanings, of gender, race, species, marketability, and cri- 
minality, are done both autonomously and invisibly; we cannot see 

how we are being named, much less audit the processes through 

which that naming happens.” 


In short, as Paglen states, “he who controls the training sets con- 
trols the meanings of images.” The naming of the categories of 
training sets ultimately determines the kinds of knowledge an algo- 
rithm can (and cannot) produce. Playing with the naming of the cor- 
puses not only allows Paglen to introduce metaphors that describe 
the contexts of the technology’s use but also exposes a loosening 
in the interplay of signs, both visual and linguistic, that is occur- 
ring through the technology. The algorithmic game that produces 
the images in Adversarially Evolved Hallucination bears on notions of 
recognition, creating a space of negotiation between what is known 
and what can be seen/recognized. Through the use of language 
and metaphor, Paglen presents not only the socio-political con- 
texts of the technology but also to suggest that human culture and 
imagination can intervene in and counter the automated naming 
processes that ascribe meaning, processes that otherwise remain 
hidden within the internal, circulatory processes of the technolo- 
gy. Through language and visual metaphor, Paglen injects cultural 
meaning and context into machine-vision processes. In doing this, 
he also highlights a central issue of A.I.: that cultural meaning and 
context are two major blind spots of machine-vision processes. 


11 Trevor Paglen, “Machine Realism,” in I Was Raised on the Internet, ed. 
Omar Kholeif (Munich: Prestel, 2018), 118. 


12 Ibid, 116. 
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Eigenface (Even the Dead Are Not Safe) (2017) 


Paglen elevates the eigenface image to the level of an artwork in his 
series Eigenface (Even the Dead Are Not Safe). His eigenface portraits 
are not formed from an average of multiple faces but rather from a 
compilation of multiple facial images of the same subject. For his 
subjects, he chose philosophical and literary figures such as Franz 
Fanon, Samuel Beckett and Simone Weil. (He also made one of the 
actress, Winona Ryder.) His process of making these eigenface 
images involved projecting eigenvectors made of each individual 
subject on to face spaces of other people and mathematically calcu- 
lating the differences between them. In this way, Paglen reenacted 
the usual recognition process of the eigenface method, but instead 
of projecting a captured image of an unknown face, he projected 
an eigenvector of a known subject, and the resulting differences, 
which are normally translated into code and stored in a database 
as representing the identity of the subject, he instead visualized in 
these images. Because of the way it depicts the features that distin- 
guish the subject from everyone else, Paglen refers to the resulting 
image as a “faceprint.”” 

As an artwork, Paglen’s eigenface portraits contrast with the ei- 
genface images that normally result from the operation of its algo- 
rithm, examples of which I have discussed in previous chapters. In 
the first instance, these portraits are in color. Rather than the usual 
gray-scale images, these portraits have a washed-out, cold, sepia 
tone. As they are compilations of multiple images of a single subject, 
these portraits also clearly portray that subject, albeit with the blur 
characteristic of eigenface images. The aesthetic of the mathemati- 
cally abstracted blur, together with the washed-out tones, produces 
a ghostly rendering of these subjects. The image of Franz Fanon, ti- 
tled Fanon (Even the Dead Are Not Safe) (figure 30), is, I find, the most 
striking of these portraits. It is a haunting portrayal of the ghost of 
Fanon, with his face veiled in a chalky pallor and the color of his skin 
only retained around the shadows of his eyes and lips and along the 
edges of his face. 


13 Metro Pictures, Trevor Paglen: A Study of Invisible Images, 9. 
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Figure 30: Trevor Paglen “Fanon” Eigenface (Even the Dead Are Not 
Safe), 2017. dye sublimation metal print. Image courtesy of the artist 


There is an expression of urgency in his return gaze. Upon staring 
into his eyes, one is unsettled by the fact that the color differs slight- 
ly in each — one with a bluish hue and the other brown. Fanon once 
stated: “We who come from the Antilles know one thing only too 
well: Blue eyes, the people say, frighten the Negro.” This portrait 
thus appears as something of a warning from the past, like Walter 
Benjamin’s take on Paul Klee’s Angelus Novus as an angel of history; 
the ghostly gaze of Fanon looks back at us from a rendering of his 
likeness created by systems of surveillance, as something to fear. 
The warning present in the Fanon image is also heralded by the 
parenthetical remark in its title (which appears in the titles of all 


14 Franz Fanon, Black Skin, White Masks, trans. Charles Lam Markmann 
(London: Pluto Press, 1986), 29. 
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the images in the series): Even the Dead Are Not Safe. Given that he is 
rendering images of historical figures, Paglen’s use of this phrase 
seems apt. Yet this phrase also conjures up a passage from Benja- 
min’s last major piece of writing, “Theses on the Philosophy of His- 
tory,” a text in which he articulates a critique of historicism and the 
method of Marxist historical materialism. To put it briefly here, 
Benjamin critiques the idea of history as a continuous path toward 
progress, and he instead articulates an understanding of history 
as a memory that arises in a moment of danger. In the same pas- 
sage, he alleges historicism has been used for the benefit of the “the 
ruling classes.” He states: “In every era the attempt must be made 
anew to wrest tradition away from a conformism that is about to 
overpower it [...] Only that historian will have the gift of fanning 
the spark of hope in the past who is firmly convinced that even the 
dead will not be safe from the enemy if he wins.” In this passage, 
Benjamin describes the ways in which the past may be articulated 
in the present and have a way of threatening both. The representa- 
tion of the figures in Paglen’s eigenface images can be understood 
to invoke Benjamin’s warning. 

The work of Fanon has been referenced in connection to cri- 
tiques of biometric systems before. In surveillance scholar Simone 
Browne’ study of contemporary surveillance practices and the 
historical archive of transatlantic slavery, she borrows from Franz 
Fanors term epidermalization in order to coin the term “digital epi- 
dermalization,” which she defines as “the exercise of power cast by 
the disembodied gaze of certain surveillance technologies [...] that 
can be employed to do the work of alienating the subject by produc- 
ing a truth about the racial body and one’s identity (or identities) 
despite the subjects’ claims.” Browne's use of the term defines a 
denied subjectivity that is produced through the application of bio- 
metric recognition. Paglen’s eigenface of Fanon visualizes this de- 
nied subjectivity with a representation of Fanon produced through 
the very systems of this disembodied gaze. As Paglen brings forth 
Fanon’s figure in this piece as a ghostly warning, an invitation of di- 


15 Benjamin, “Theses on the Philosophy of History,” 255 (italics in original). 


16 Browne, Dark Matters, 110. 
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alogue between Fanon’s discourse and the dialectics of recognition 
emerges. 


Figure 31: Trevor Paglen, “Beckett” Eigenface (Even the Dead Are Not 
Safe), 2017. dye sublimation metal print. Image courtesy of the artist 


The portraits of Beckett and Weil (figure 31 and figure 32), like the 
Fanon portrait, present their subjects with a neutral expression and 
in a forward-facing pose. Fixed as an eigenface image, the head 
of each figure floats ominously in a sea of pixelated blur. What to 
make of these ghostly heads of dead revolutionaries and philoso- 
phers? As with the production of his images in the Adversarial Hallu- 
cinations series, with his eigenface portraits Paglen customizes the 
training set category; he chooses his subjects, all of whom stand out 
as archetypes of critical thinkers. Weil’s face, however, wears the 
expression of a slight, Mona Lisa-like smile. This is perhaps because, 
as a woman, she has been expected to smile and to generally present 
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a pleasant facial expression that is not expected of men. These are, 
after all, statistical averages of a collection of facial images of each 
figure, and as such they capture something of the public expression 
of each subject. 


Figure 32: Trevor Paglen, “Weil” Eigenface (Even the Dead Are Not Safe), 
2017. dye sublimation metal print. Image courtesy of the artist 


In their eigenface images, these three figures are transformed into 
archetypes in two ways. They are each archetypes of a particular 
kind of critical thinker. The work of each offers critical analyses of 
systems of power that speak to the contexts of facial recognition 
and its processes of meaning production. Beckett, for example, in 
his work as a novelist, playwright and poet, represents an attack on 
the realist tradition; he dispenses with the narrative element of a 
unity of time and place, instead focusing on revealing the essence 
of the human condition. Weil’s work as a philosopher and as a polit- 
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ical activist fighting for the interests of the working class included 
a critique of forms of oppression, whether the exploitation of capi- 
talism or the elitist bureaucracy of Marxism. The eigenface images 
of each present a visual rendering of each figure as an archetype 
of critical thought. Yet, through the gender and race of each figure, 
these portraits also present archetypes of institutional categories 
of identification, an association I think is intentional on the artist’s 
part. As such, we can surmise that “even the dead are not safe,” that 
is, not safe from their inclusion within a categorical “type” as part of 
the identification practices of AFR technology. 

In producing portraits of subjects whose work precedes their 
image, and whose work also contributes to a critique of the notions 
of power and identity, Paglen inverts a truth of recognition through 
automated processes. In bringing these figures “back from the dead” 
through the form of an eigenface classifier, he brings together the 
discourses of critical thought in philosophy and contemporary sys- 
tems of institutional enforcement. The critiques of power that are 
embodied in these figures are tools. In the course of a discussion of 
algorithmic prediction in facial recognition, Paglen mentions: “An 
analysis of power, of capital, of race, [these] are the kinds of tools 
that you bring to the conversation that are ultimately more help- 
ful.”” Of the eigenface portraits, he states: “someone like a Fanon 
or a Weil contributed to social progress precisely by breaking the 
law — because they were unjust laws [...] It’s asking whether the de- 
velopment of these technologies will preclude people like Simone 
Weil or Franz Fanon from ever existing again.” Paglen’s question is 
whether, in a society that is increasingly organized through meta- 
data and predictive systems and, as such, enforces certain kinds 
of “normative behavior,” any revolutionary thought and action can 
survive. Paglen’s engagement with this problem involves reference 
to historical figures and thereby speaks to a precedent of the tech- 
nology. In other words, there is an element of answering the future 


17 Charlie Robin Jones, “The Artist Trevor Paglen, The Surveillance State, and 
Your Life,” SSENSE, May 15, 2019, https://www.ssense.com/en-us/edito 
rial/art/this-artist-the-military-industrial-complex-and-your-life. 


18 Ibid. 
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(both the predictive aspect of the algorithm and a near-future soci- 
ety of increasing automation) with the past. Paglen intentionally ex- 
ploits the “ghost-face”” character of eigenface images as an aesthet- 
ic in these portraits, as a way of communicating how the subjects 
and their legacies of thought haunt the present. Paglen conjures up 
these figures not only as ominous reminders of the unequal power 
structures that underlie facial recognition technology but also as a 
reference to the present necessity of tools of critical discourse. 


Concluding Remarks 


Through both Adversarially Evolved Hallucination (2017) and Eigenface 
(Even the Dead Are Not Safe) (2017), Paglen gets inside some of the is- 
sues at the heart of the design of automated recognition technology. 
Where other scholars and artists have scrutinized the normative 
categories that guide facial recognition technologies, Paglen chal- 
lenges how these normative categories are constructed through the 
technology. In appropriating the training-set phase of the image 
recognition algorithm, Paglen arrives at one of the ways in which 
the technology ascribes meaning in the world. Through his imagery 
he creates a space for dialogue between the dialectics of recogni- 
tion and the modes of discourse in critical thinking. Together, these 
approaches situate the technological developments of machine 
vision technologies within a framework of cultural and political 
thought. In doing this, he also articulates an existing gap between 
the design and development of machine-vision systems and the cul- 
tural and political worlds in which they are deployed. His images 
suggest to the observer that the implications of the technology are 
open ended and as images, they are open to interpretation. As such, 
he highlights an inherent fluidity of human perception in the face 
of the machinic gaze. 


19 Carikci and Özen, “A Face Recognition System,” 122. 
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Contribution to the field 


This study has explored the technology of automated facial rec- 
ognition through the discourse of visual culture studies and con- 
temporary artistic practice. This analysis addresses one of the 
central issues of contemporary visuality, that is, the technological 
advancement of forms of machinic vision through AFR technology 
and the associated shift in our understandings of what it is to see 
and be seen. The discussion has addressed a general problematic 
of automated recognition and framed its enactment as a technical, 
cultural and philosophical process. A primary theoretical aim has 
been to approach the recognition process of AFR technology as an 
example of machinic vision, that is, a mode of perception involv- 
ing a disembodied perspective, and to analyze how it operates in 
contemporary society. Taking the processes of an AFR method as 
its point of departure, this study has shown how machinic vision is 
technically defined through an automated operation of recognition. 
Ithas, further, explored a history in which statistics and vision have 
been intimately connected, revealing this connection to be an or- 
ganizing principle of AFR processes and a representational mecha- 
nism through which automated recognition occurs. Contemporary 
examples of artistic practice have provided further insight into the 
contexts of AFR implementation and suggested new approaches to 
the conceptualization of this technology. As I have shown, the art- 
works included in this study also actively reimagine the processes 
placing them in dialogue with cultural and socio-political contexts 
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broadening our understanding of the dialectics of recognition in a 
machine-vision process. 

Through its exploration of the shifting role of the image from 
pictorial to algorithmic with the advent of digital networks, this 
analysis contributes to the field of visual culture studies.' Technol- 
ogists have argued that machines do not see — that AFR technology 
does not relate to historical discourses on visuality or photography 
but rather introduces an entirely new digital landscape in which 
all visual input, whether pictorial or coded, is and should be con- 
sidered as data. While taking this view into account, I argue that 
contemporary digital technologies and the infrastructures in which 
they operate do not present an entirely new world but rather car- 
ry within them historical continuities of cultural and visual logic. 
With reference to concepts such as the technical image and the 
operational image,” notions that track this shift in the role of im- 
ages as information, I have approached the eigenface image as an 
object of inquiry. The eigenface image provides an image of data 
and, as such, serves as a meeting point for human perception and 
algorithmic perception. I have used the example of eigenface to 
reflect on this tension between image and data. This analysis does 
not confound images with data but rather attempts to clarify the 
complex relationship between the two. In relating the representa- 
tional mechanisms of AFR technology to the historical practice of 
composite portraiture within the sciences, philosophy and art, I 
have explained the way this technology relies on a cultural logic that 
involves the merging of statistics with vision. By tracing this statis- 
tical way of seeing, I have brought to light the nuanced relationship 
between images and data, rather than simply suggesting that there 
is a mutually exclusive relationship between these two categories. 

This analysis also makes a methodological contribution in its 
departure from traditional approaches within the field of visual 


1 For more on this topic see Hito Steyerl, “A Sea of Data: Apophenia and Pat- 
tern (Mis-)Recognition,” e-flux Journal 72 (April 5, 2016), https://www.e-f 
Zlux.com/journal/72/60480/a-sea-of-data-apophenia-and-pattern-mis-re 
cognition/. 


2 Vilem Flusser, Into the Universe, 10; Farocki, War at a Distance. 
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culture: namely, it expands the scope of analysis beyond the image 
to include the processes by which the image is produced. In the con- 
text of analyzing machinic vision, this is imperative if we are to un- 
derstand the ways in which the image operates and to reveal, inter- 
rogate and critique the conditions of algorithmic vision. Referring 
to the work of Jonathan Crary’ and his analysis of the historical con- 
struction of vision through the position of the observer, this study 
of a particular AFR method has attempted to explain how its tech- 
nology embodies a machinic observer, and to explore the historical 
continuities and discontinuities present in this new kind of visual 
organization. This study has shown how an analysis of the visual- 
ity of AFR technology can contribute to a broader understanding 
of the varying modes of perception found in contemporary society 
and their roles in intervening across social, political and cultural 
terrains. 

This analysis joins certain other scholarly works in critiquing 
the limitations of AFR technology from a perspective in the hu- 
manities.* Drawing on theoretical perspectives from surveillance 
studies, gender and race studies, and media and communications 
studies, these previous scholarly works have not approached the 
subject matter from the standpoint of the sciences, that is, from 
within the fields in which these technologies have been developed, 
but from beyond them, in order to achieve more expansive notions 
about what counts as the success or failure of biometric technolo- 
gies and to understand their cultural, political and social implica- 
tions. Similarly, this analysis approaches AFR technology as a form 
of machinic vision by bringing its processes into a dialogue with a 
theoretical framework premised on visual culture and its associat- 
ed political, cultural and philosophical ideas. 

The findings of this analysis support a central critique leveled 
by the authors of many of these previous works: that recognition 
by an AFR system enforces normative categories of institutional- 
ly based identity in the service of ever-tighter biopolitical control. 
This is analyzed through the discussion on the ways of seeing em- 


3 Crary, Techniques of the Observer. 


4 See discussion in Cultural Analysis of Biometrics: Previous Scholarship, p. 22. 
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bedded in the eigenface method and its reductive representational 
mechanisms. In an operation of recognition carried out by an AFR 
system, any understanding of identity that is both subjective and 
fluid gets negated. At a time when identity politics is in the ascen- 
dant and self-generated notions of identity are crucial to political 
agency, AFR systems automate the enforcement of institutionally 
based identities. This enforcement of normative categories of iden- 
tity has the potential to disproportionately affect those groups that 
are already vulnerable and marginalized in relation to systems of 
governance including the distribution of state services, border con- 
trol and policing — three key contexts in which AFR is increasingly 
being employed. For AFR systems are not only implemented in con- 
texts of security and risk mitigation; they are also used to grant or 
deny access to resources. The negation of recognition may be just as 
dangerous as being unwittingly registered in a biometric database. 
This study of the machinic ways of seeing used in a particular AFR 
method of eigenface has revealed a history of the enforcement of 
institutionally based identities; these identities are further opera- 
tionalized through these automated technologies of recognition. 


Revisiting the Aim: Looking Back 


In sum, the problem with AFR and its machinic way of seeing is that 
it applies a statistical and thereby reductive method of recognition 
to a subject whose identity cannot ultimately be measured, defined 
or known by these means. This study began with three research 
questions, the first of which asked how recognition is defined in 
an AFR method. The analysis of the eigenface algorithm in Part I 
explored three aspects that constitute recognition and perception 
by algorithm: the statistical pattern recognition method of Princi- 
pal Component Analysis (PCA), the production of facial aggregates 
called eigenvectors and the formation of the subspace called the 
face space. The eigenface method recognizes individual faces by 
differentiating them from a statistical average. The primary repre- 
sentational mechanism of eigenface, that is, the algorithm’s means 
of coming to know and recognize faces, relies on a statistical meth- 
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od. Recognition through the eigenface method occurs through the 
merging of statistics with vision. This merging of statistics with vi- 
sion as a means of recognition has various ontological implications. 
One of these was explored in this thesis in the discussion of the 
historical antecedent of the eigenface method: Galton’s composite 
portraiture and his method of using statistical representations of 
faces to uncover sociological types. The discussion of Galton’s ap- 
proach to composite portraiture responded to the second research 
question of this study, on the historical continuities and discon- 
tinuities between AFR methods and their antecedents. With this 
account of Galton’s composite-portrait practice, this study empha- 
sized the cultural and historical context of the origins of a merging 
of statistics and vision, namely the way Galton’s broader project of 
eugenics and the visualization of reductive, normative sociological 
categories inform a particular statistical way of seeing. 

An alternative to this statistical way of seeing - and of inter- 
preting the significance of the composite portrait — is provided by 
Ludwig Wittgenstein, who puts forward an alternative take on the 
meaning of the composite’s aesthetic form. Approaching the com- 
posite portrait as an object of study within the field of philosophy 
and within his project of investigating language games, Wittgen- 
stein argues that we should recognize that the composite portrait 
contains multiple overlapping forms. Rather than perceiving the 
composite as an expression of statistical reduction, he sees it as an 
expression of a perceptual fluidity. Through his concept of aspect 
perception, Wittgenstein describes a perceptual ability to perceive 
forms shifting into other forms. I argue that, in his approach to the 
composite portrait, Wittgenstein provides an inherently utopian 
vision that expresses the kind of fluidity found in current discours- 
es of identity politics and the politics of representation, a vision that 
resists the reductive processes of recognition through AFR systems 
and their utilization of biopolitical control. 

Wittgenstein’s alternative take on the composite portrait was 
expanded upon in Part II, which explored the artistic engagements 
of contemporary artists with the technology of facial recognition. 
This brings us to the third and final research question of this study, 
which asked how contemporary artists have articulated and prob- 
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lematized the cultural implications of AFR technology. I discussed 
examples of contemporary artistic practice in order to provide fur- 
ther insight into the contexts of AFR implementation and suggest 
new ways of conceiving of this technology. Each artist presents an 
engagement with facial recognition technology within a particular 
cultural discourse, thus answering a central question posed by this 
technology: what meaning can be derived through a process of au- 
tomated facial recognition? 


The Composite Form 


Throughout this study, the motif of the composite portrait has ap- 
peared in different forms. The composite emerges as an aesthetic 
form that is expressive of how information is structured. This in- 
vestigation has shown that the composite form, as it has expressed 
itself in practices of facial recognition, has two diverging and 
contradictory meanings that are developed through two different 
discourses. In the analysis of eigenface, the digital composite was 
introduced as the phantom face that peers out from the eigenface 
image, as the visualization of a statistically constructed pattern of 
multiple faces. In Francis Galton’s work, the composite portrait was 
deployed in the attempt to produce images of sociological types. 
Although ultimately failing in this endeavor, Galton’s composite 
portraits were an original attempt to give visibility to a knowledge 
only possible and operational through statistics and to make this 
knowledge accessible to the human eye. The eigenface image and 
Galton’s composite portrait are examples of the composite form 
being utilized within the areas of science and governance and op- 
erationalized as a predictive tool through reductive logic. But in 
Wittgenstein’s perceptual approach to the composite portrait — now 
transplanted to the discourse of philosophy — it is an expression of 
multiplicity in which all the phenomena that make up the compos- 
ite lie open to view.” The composite portrait is an expression of a 
relationship between varying forms. The possibilities of meaning 


5 Wittgenstein, Philosophical Investigations, 126. 
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contained within the perception of the composite form also, on this 
view, multiply. This study concludes that the composite form can 
be expressive of an entirely different and contrary meaning from 
the one suggested by the reductive, statistical interpretation of the 
form. 

This alternative approach to the composite form was further 
explored in Part II. Through a discussion of works by three artists, 
Thomas Ruff, Zach Blas and Trevor Paglen, I examined the varied 
expressions of the composite motif in contemporary artworks en- 
gaging with the theme of facial recognition. The artworks in this 
study extract the composite and the structures of AFR knowledge 
from the discourses of science and governance, and they explore 
the myriad representational uses of the composite. In Ruff’s series 
of andere Portraits, silk-screened photographic composite mugshots 
explore the notion that the act of representation itself constructs 
an identity. The binary and analogue composite form used in this 
series of portraits is such that one’s eyes are never able to rest on 
a single face. Like a kind of large-scale visualization of Wittgen- 
stein’s concept of aspect perception, these portraits express a con- 
stant fluidity — between faces and, also, between notions of gender. 
Blas’s sculptural masks, in his Facial Weaponization Suite, transform 
composite portraiture into physical form. Blas’s sculptural compos- 
ites reference a notion of excess: that is, the multiplicity of facial 
forms that constitute his masks counter and resist the singularity 
of a recognition operation by an AFR system. A key contribution 
of Blas’s work is his use of the form of the composite, that is, the 
aesthetic form that structures the information used by an AFR sys- 
tem, to counter the very operation of this technology. Drawing on 
discourses in post-colonial theory and theories of political agency, 
Blas uses his masks to reassert an unruliness in representations of 
the body. His work also points towards a notion of the subject as 
composite, through a political strategy of collectivism. Paglen’s use 
of the composite form in two series, Adversarial Hallucinations and 
Eigenface (Even the Dead Are Not Safe), appropriates the categories of 
knowledge used to train AI systems of recognition. By presenting 
algorithmic-based imagery, his composites reveal a relationship be- 
tween training data and an algorithmic ability not only to recognize 
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but to reimagine physical form. Paglen’s use of literary metaphors 
and ideological concepts to categorize training data speaks to the 
deeper truths that structure contemporary surveillance systems 
and offers a means of critiquing their cultural logic. 

Through these alternative explorations of the composite form 
and facial recognition, these artworks reassert a visual field that is 
fluid, open, mutable and dynamic. As such, they are able to reveal 
the limitations of a machinic way of seeing and instead put forward 
a way of seeing that can measure up to the truth of the recognized 
subject. As Jenny Edkins states, “the contemporary face — the face 
of biopolitics and surveillance — forces a being into presence, but 
into presence as an object not a person. The being, in all its mys- 
tery, its unknowability, is missing. As well as providing these 
expressions of an open, mutable perception, these artworks allow 
for a measure of unknowability in relation to the subject. This space 
allows a subject to choose her own identity or identities, based on 
an embodied, subjective experience, and also allows the subject to 
have the right to reject enforced, institutional categories of iden- 
tity. This right continues in its importance because institutional 
identities are constructed by institutions of governance — the same 
institutions of governance that have historically deemed black and 
brown people sub-human, cast Jews as the enemy and today detain 
a growing population of migrants in detention camps, defining 
these groups as stateless and thereby without access to certain, in- 
alienable rights. 


Recommendations: Looking Forward 


During the final stages of writing this book, the New York Times 
carried a news report about the first known use of AFR by a gov- 
ernment for the purposes of racial profiling.’ The report described 
the use of AI by authorities in China, in particular, the use of facial 


6 Edkins, Face Politics, 7. 


7 Paul Mozur, “One Month, 500,000 Face Scans: How China Is Using AI to 
Profile a Minority,” The New York Times, April 14, 2019. https://www.nytimes. 
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recognition technology to surveil and track members of the Uighur 
community, a minority ethnic Muslim population, for the purposes 
of law enforcement.® The report explains that recognition is strict- 
ly based on appearance, as Uighurs look distinct from the major- 
ity Han population in China, which makes it “easier for software 
to single them out.” The report describes the creation by Chinese 
authorities of face-image databases not only for Uighurs but also 
“for people with criminal records, mental illnesses, records of drug 
use and those who petitioned the government over grievances.”°In 
light of the historical analysis offered in this study, we may say that, 
with the use of AFR technology in China, history is repeating itself. 
My analysis of the facial representation mechanisms found within 
the eigenface algorithm and the practice of differentiating facial 
groups from a “norm” makes it clear that the possibility of using 
this technology for racial profiling purposes is built into the very 
ways of seeing that underlie AFR processes. The report on the use of 
AFR by the Chinese authorities also describes a shift in the develop- 
ment of AI technologies; where before democratic societies had the 
upper hand in this sector, authoritarian regimes are now producing 
a new generation of tech startups that cater to their needs." This 
authoritarian use of AFR processes not only reflects one possible 
path of future development for this technology but also recalls a 
history in the discourses and practices of the politics of represen- 
tation. In conclusion, this analysis recommends looking toward the 
aesthetic forms that structure algorithmic knowledge as sources 
of inquiry and contestation. In doing so, it has engaged with a his- 
torical discourse on the relationship between visual perception and 
knowledge, and it has sought to relate this to an alternative way in 
which technology might define and organize visual perception in 
the future. 


com/2019/04/14/technology/china-surveillance-artificial-intelligence-raci 
al-profiling.html?partner=IFTTT. 

8 Ibid. 

9 Ibid. 

10 Ibid. 

11 Ibid. 
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This book began with a description of an experience I had of 
being biometrically registered at an airport border crossing, and it 
ends with this report of the first known use of AFR for racial pro- 
filing purposes. The growing use of AFR in a variety of areas, from 
risk mitigation and the control of people’s movements and access to 
resources to the profiling of vulnerable groups, indicates an inten- 
sification of the interconnections between its automated processes 
and contemporary notions of recognition, representation and iden- 
tity. It also indicates the necessity of approaching algorithmic pro- 
cesses as possible objects of critique, of problematizing their logic 
and dissecting and seeing through their everyday use. This means 
ensuring that the processes and narratives of algorithms - their 
weaknesses, their logic and their implications - are transparent 
to us. This analysis joins those of other scholars in the field of the 
humanities who have found it necessary, and possible, to question 
the logic of algorithms and to dissect their use. The issues that arise 
from the technological development of AI and the application of 
AFR reveal that contemporary shifts in technology go hand in hand 
with a shift in culture. In the face of our increasing dependence on 
AFR technology, this visual culture analysis of the phenomenon has, 
in sum, amounted to an academic version of an A.I model of always 
keeping the “human in the loop” - by keeping the humanities in the 
loop and recognizing that they are, almost always, already in the 
loop. 
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Trevor Paglen, “Highway of Death (Corpus: The Aftermath 
of the First Smart War)” Adversarially Evolved Hallucination, 
2017. dye sublimation metal print. Image courtesy of the 
artist. 


Trevor Paglen, “Vampire (Corpus: Monsters of Capitalism)”. 
Adversarially Evolved Hallucination, 2017. dye sublimation 
metal print. Image courtesy of the artist. 


Trevor Paglen, training set images for “Vampire (Corpus: 
Monsters of Capitalism)” Adversarially Evolved Hallucina- 
tion, 2017. Image courtesy of the artist. 


Trevor Paglen “Fanon” Eigenface (Even the Dead Are Not Safe), 
2017. dye sublimation metal print. Image courtesy of the 
artist. 


Trevor Paglen, “Beckett” Eigenface (Even the Dead Are Not 
Safe), 2017. dye sublimation metal print. Image courtesy of 
the artist. 


Trevor Paglen, “Weil” Eigenface (Even the Dead Are Not Safe), 
2017. dye sublimation metal print. Image courtesy of the 
artist. 


